2024-04-06T07:41:00.968 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-06T07:41:00.969 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} --name teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392 -- /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/orig.config.yaml 2024-04-06T07:41:01.020 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T07:41:01.114 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392 branch: reef description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: ceph-qa@ceph.com first_in_suite: false job_id: '7637392' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9400 sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 sleep_before_teardown: 0 subset: 12/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 052adcadd0534eddb0d73c5afbc083a84cd9dc46 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTHMeTm4aEYxXTztGJp/ZnBWzFTNmzWr6I2B05DKL+lPIWvSFoRnUfJ4TA4EGUpMSupoLkseaDMlft5b+aCxmc= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEK2+aO2cc17S44L4cN0L8KoZ3ZphY9k5HggEzkX1MzM4Z8bL4MNfCMihKsDKb7zUo6rjGLJC0m70T94BwUN50= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_22:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-06T07:41:01.114 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa; will attempt to use it 2024-04-06T07:41:01.115 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks 2024-04-06T07:41:01.115 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T07:41:01.117 INFO:teuthology.task.internal:Checking packages... 2024-04-06T07:41:01.140 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4d05f4cad464de0fcdd11a2856643e533aea34e8' 2024-04-06T07:41:01.141 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T07:41:01.141 INFO:teuthology.packaging:ref: None 2024-04-06T07:41:01.141 INFO:teuthology.packaging:tag: None 2024-04-06T07:41:01.141 INFO:teuthology.packaging:branch: reef 2024-04-06T07:41:01.141 INFO:teuthology.packaging:sha1: 4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T07:41:01.141 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-04-06T07:41:01.440 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1169-g37f5b3f8-1jammy 2024-04-06T07:41:01.441 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T07:41:01.449 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T07:41:01.449 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T07:41:01.459 INFO:teuthology.task.internal:Saving configuration 2024-04-06T07:41:01.476 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T07:41:01.482 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T07:41:01.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 07:35:17.078725', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN3ftWcNkj+AeZUBgZwb0e1Qc/YgnBcC93nOBwZi/Wch'} 2024-04-06T07:41:01.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 07:35:17.080573', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO04wtEMz6Ao6rtNyxinn63+CKaQojITTPU9H+XgMeM0'} 2024-04-06T07:41:01.545 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T07:41:01.552 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-04-06T07:41:01.552 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-04-06T07:41:01.552 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T07:41:01.651 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f038968e040>, signals=[15]) 2024-04-06T07:41:01.651 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T07:41:01.657 INFO:teuthology.task.internal:Opening connections... 2024-04-06T07:41:01.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T07:41:01.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:41:01.750 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:41:01.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:41:01.836 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T07:41:01.842 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-04-06T07:41:01.848 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-04-06T07:41:01.848 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Ubuntu" 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="22.04" 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_CODENAME=jammy 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:ID=ubuntu 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE=debian 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-06T07:41:01.896 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-06T07:41:01.897 INFO:teuthology.orchestra.run.smithi047.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-06T07:41:01.897 INFO:teuthology.orchestra.run.smithi047.stdout:UBUNTU_CODENAME=jammy 2024-04-06T07:41:01.897 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-04-06T07:41:01.928 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-04-06T07:41:01.933 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-04-06T07:41:01.933 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Ubuntu" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="22.04" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_CODENAME=jammy 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:ID=ubuntu 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE=debian 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-06T07:41:01.981 INFO:teuthology.orchestra.run.smithi129.stdout:UBUNTU_CODENAME=jammy 2024-04-06T07:41:01.982 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-04-06T07:41:02.020 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T07:41:02.029 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T07:41:02.050 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T07:41:02.050 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-04-06T07:41:02.053 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-04-06T07:41:02.058 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T07:41:02.064 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T07:41:02.064 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-04-06T07:41:02.101 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-04-06T07:41:02.109 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T07:41:02.183 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T07:41:02.197 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-06T07:41:02.197 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-06T07:41:02.198 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T07:41:02.198 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T07:41:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:41:02.203 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-04-06T07:41:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:41:02.204 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-04-06T07:41:02.249 INFO:teuthology.orchestra.run.smithi047.stdout:5.15.0-56-generic 2024-04-06T07:41:02.249 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.15.0-56-generic 2024-04-06T07:41:02.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get clean 2024-04-06T07:41:02.250 INFO:teuthology.orchestra.run.smithi129.stdout:5.15.0-56-generic 2024-04-06T07:41:02.250 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.15.0-56-generic 2024-04-06T07:41:02.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2024-04-06T07:41:02.378 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-04-06T07:41:02.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-04-06T07:41:02.561 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T07:41:02.561 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-06T07:41:02.571 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-06T07:41:02.654 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T07:41:02.655 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-06T07:41:02.688 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-06T07:41:02.715 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-06T07:41:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-06T07:41:02.868 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-06T07:41:02.902 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-06T07:41:02.907 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-06T07:41:02.909 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-06T07:41:02.909 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-06T07:41:02.910 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-06T07:41:02.934 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-06T07:41:02.937 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T07:41:02.937 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-06T07:41:02.944 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-06T07:41:02.954 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-06T07:41:02.955 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-06T07:41:02.957 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-06T07:41:02.972 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-06T07:41:02.972 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-06T07:41:02.978 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-06T07:41:02.978 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-06T07:41:02.985 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-06T07:41:02.989 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-06T07:41:02.989 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,648 kB] 2024-04-06T07:41:03.011 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.2 kB] 2024-04-06T07:41:03.011 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [275 kB] 2024-04-06T07:41:03.015 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T07:41:03.015 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-06T07:41:03.025 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-06T07:41:03.031 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-06T07:41:03.034 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-06T07:41:03.035 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-06T07:41:03.035 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-06T07:41:03.053 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-06T07:41:03.053 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-06T07:41:03.053 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-06T07:41:03.054 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-06T07:41:03.055 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-06T07:41:03.055 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-06T07:41:03.075 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-06T07:41:03.076 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-06T07:41:03.076 INFO:teuthology.orchestra.run.smithi129.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-06T07:41:03.076 INFO:teuthology.orchestra.run.smithi129.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-06T07:41:03.080 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-06T07:41:03.080 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-06T07:41:03.218 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-06T07:41:03.370 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-06T07:41:03.414 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-06T07:41:03.536 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-06T07:41:03.547 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-06T07:41:03.548 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-06T07:41:03.550 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-06T07:41:03.560 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-06T07:41:03.575 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-06T07:41:03.576 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.2 kB] 2024-04-06T07:41:03.577 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,648 kB] 2024-04-06T07:41:03.641 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-06T07:41:03.641 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [275 kB] 2024-04-06T07:41:03.644 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T07:41:03.644 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-06T07:41:03.646 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T07:41:03.661 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-06T07:41:03.810 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-06T07:41:03.839 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-06T07:41:04.238 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-06T07:41:04.244 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-06T07:41:04.258 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-06T07:41:04.259 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-06T07:41:04.261 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-06T07:41:04.434 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-06T07:41:04.472 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-06T07:41:04.484 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-06T07:41:04.486 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-06T07:41:04.486 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-06T07:41:04.489 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-06T07:41:04.489 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-06T07:41:04.490 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-06T07:41:04.516 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-06T07:41:04.519 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-06T07:41:04.683 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-06T07:41:04.765 INFO:teuthology.orchestra.run.smithi047.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-06T07:41:04.850 INFO:teuthology.orchestra.run.smithi047.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-06T07:41:04.930 INFO:teuthology.orchestra.run.smithi047.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-06T07:41:04.961 INFO:teuthology.orchestra.run.smithi047.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-06T07:41:08.543 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 12.6 MB in 2s (7,046 kB/s) 2024-04-06T07:41:09.474 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 12.6 MB in 3s (5,012 kB/s) 2024-04-06T07:41:09.963 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-04-06T07:41:09.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T07:41:10.032 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-04-06T07:41:10.209 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-04-06T07:41:10.210 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-04-06T07:41:10.363 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T07:41:10.363 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T07:41:10.363 INFO:teuthology.orchestra.run.smithi129.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T07:41:10.364 INFO:teuthology.orchestra.run.smithi129.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T07:41:10.364 INFO:teuthology.orchestra.run.smithi129.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-06T07:41:10.364 INFO:teuthology.orchestra.run.smithi129.stdout: python2.7-minimal 2024-04-06T07:41:10.364 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T07:41:10.365 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2024-04-06T07:41:10.365 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T07:41:10.365 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-06T07:41:10.365 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-06T07:41:10.366 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2024-04-06T07:41:10.366 INFO:teuthology.orchestra.run.smithi129.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-06T07:41:10.366 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2024-04-06T07:41:10.367 INFO:teuthology.orchestra.run.smithi129.stdout: thermald 2024-04-06T07:41:10.414 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2024-04-06T07:41:10.414 INFO:teuthology.orchestra.run.smithi129.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T07:41:10.414 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-06T07:41:10.414 INFO:teuthology.orchestra.run.smithi129.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-06T07:41:10.415 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages will be upgraded: 2024-04-06T07:41:10.416 INFO:teuthology.orchestra.run.smithi129.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-06T07:41:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T07:41:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 113 MB of archives. 2024-04-06T07:41:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-06T07:41:10.595 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-06T07:41:10.769 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-04-06T07:41:10.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T07:41:10.839 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-04-06T07:41:11.018 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-04-06T07:41:11.018 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-04-06T07:41:11.176 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T07:41:11.176 INFO:teuthology.orchestra.run.smithi047.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T07:41:11.177 INFO:teuthology.orchestra.run.smithi047.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T07:41:11.177 INFO:teuthology.orchestra.run.smithi047.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T07:41:11.177 INFO:teuthology.orchestra.run.smithi047.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-06T07:41:11.177 INFO:teuthology.orchestra.run.smithi047.stdout: python2.7-minimal 2024-04-06T07:41:11.177 INFO:teuthology.orchestra.run.smithi047.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T07:41:11.178 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2024-04-06T07:41:11.178 INFO:teuthology.orchestra.run.smithi047.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T07:41:11.178 INFO:teuthology.orchestra.run.smithi047.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-06T07:41:11.178 INFO:teuthology.orchestra.run.smithi047.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-06T07:41:11.179 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2024-04-06T07:41:11.179 INFO:teuthology.orchestra.run.smithi047.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-06T07:41:11.179 INFO:teuthology.orchestra.run.smithi047.stdout:Recommended packages: 2024-04-06T07:41:11.180 INFO:teuthology.orchestra.run.smithi047.stdout: thermald 2024-04-06T07:41:11.230 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2024-04-06T07:41:11.230 INFO:teuthology.orchestra.run.smithi047.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T07:41:11.230 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-06T07:41:11.231 INFO:teuthology.orchestra.run.smithi047.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-06T07:41:11.232 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages will be upgraded: 2024-04-06T07:41:11.233 INFO:teuthology.orchestra.run.smithi047.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-06T07:41:11.290 INFO:teuthology.orchestra.run.smithi047.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T07:41:11.290 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 113 MB of archives. 2024-04-06T07:41:11.290 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-06T07:41:11.290 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-06T07:41:11.848 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-06T07:41:11.988 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-06T07:41:12.016 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-06T07:41:12.366 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-06T07:41:12.900 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-06T07:41:12.900 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-06T07:41:12.900 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-06T07:41:13.034 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-06T07:41:13.138 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-06T07:41:13.565 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 113 MB in 2s (59.6 MB/s) 2024-04-06T07:41:13.750 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-06T07:41:14.591 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-06T07:41:14.592 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-06T07:41:14.592 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-06T07:41:15.375 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-06T07:41:15.497 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-06T07:41:15.750 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-06T07:41:15.754 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:15.920 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 113 MB in 5s (22.3 MB/s) 2024-04-06T07:41:16.030 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:16.105 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-06T07:41:18.065 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-06T07:41:18.083 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-06T07:41:18.086 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:18.090 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:18.159 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:18.168 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:18.652 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-06T07:41:18.672 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:18.710 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:20.311 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-06T07:41:20.333 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:20.429 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:20.890 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-06T07:41:20.910 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:20.947 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:25.298 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T07:41:25.405 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T07:41:25.730 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T07:41:25.858 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T07:41:26.195 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-06T07:41:26.218 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-06T07:41:26.278 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T07:41:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T07:41:27.847 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T07:41:28.147 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T07:41:28.314 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T07:41:28.629 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-06T07:41:28.651 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-06T07:41:28.695 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T07:41:31.494 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-06T07:41:31.531 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:31.569 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:34.490 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T07:41:34.615 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T07:41:34.711 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-06T07:41:34.748 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T07:41:34.794 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:34.966 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T07:41:35.095 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:35.221 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-06T07:41:35.350 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:36.699 INFO:teuthology.orchestra.run.smithi047.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-06T07:41:36.699 INFO:teuthology.orchestra.run.smithi047.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-06T07:41:36.699 INFO:teuthology.orchestra.run.smithi047.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-06T07:41:36.700 INFO:teuthology.orchestra.run.smithi047.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-06T07:41:36.852 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:37.513 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T07:41:37.638 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T07:41:37.966 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T07:41:38.751 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:38.770 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:38.903 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-06T07:41:39.032 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:40.188 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-06T07:41:40.288 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-06T07:41:40.334 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-06T07:41:40.334 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-06T07:41:40.334 INFO:teuthology.orchestra.run.smithi129.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-06T07:41:40.334 INFO:teuthology.orchestra.run.smithi129.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-06T07:41:40.390 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:40.448 INFO:teuthology.orchestra.run.smithi047.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-06T07:41:40.449 INFO:teuthology.orchestra.run.smithi047.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-06T07:41:40.476 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:41.884 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:43.261 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-06T07:41:43.387 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-06T07:41:43.513 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T07:41:43.574 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-06T07:41:43.574 INFO:teuthology.orchestra.run.smithi129.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-06T07:41:55.209 INFO:teuthology.orchestra.run.smithi047.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-06T07:41:55.209 INFO:teuthology.orchestra.run.smithi047.stdout:Sourcing file `/etc/default/grub' 2024-04-06T07:41:55.219 INFO:teuthology.orchestra.run.smithi047.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T07:41:55.227 INFO:teuthology.orchestra.run.smithi047.stdout:Generating grub configuration file ... 2024-04-06T07:41:55.479 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T07:41:55.496 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T07:41:55.787 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T07:41:55.793 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T07:41:55.862 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T07:41:55.867 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T07:41:56.041 INFO:teuthology.orchestra.run.smithi047.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T07:41:56.041 INFO:teuthology.orchestra.run.smithi047.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T07:41:56.041 INFO:teuthology.orchestra.run.smithi047.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T07:41:56.086 INFO:teuthology.orchestra.run.smithi047.stdout:done 2024-04-06T07:41:56.490 DEBUG:teuthology.orchestra.run.smithi047:> dpkg -s linux-image-generic 2024-04-06T07:41:56.523 INFO:teuthology.orchestra.run.smithi047.stdout:Package: linux-image-generic 2024-04-06T07:41:56.523 INFO:teuthology.orchestra.run.smithi047.stdout:Status: install ok installed 2024-04-06T07:41:56.523 INFO:teuthology.orchestra.run.smithi047.stdout:Priority: optional 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Section: kernel 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Installed-Size: 21 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: amd64 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Source: linux-meta 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Version: 5.15.0.101.98 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Recommends: thermald 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout:Description: Generic Linux kernel image 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: This package will always depend on the latest generic kernel image 2024-04-06T07:41:56.524 INFO:teuthology.orchestra.run.smithi047.stdout: available. 2024-04-06T07:41:56.525 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-06T07:41:56.525 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T07:41:56.525 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T07:41:56.525 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-04-06T07:41:56.525 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T07:41:56.525 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get clean 2024-04-06T07:41:56.705 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-04-06T07:41:56.855 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T07:41:56.856 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T07:41:56.866 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T07:41:56.885 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T07:41:57.842 INFO:teuthology.orchestra.run.smithi129.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-06T07:41:57.842 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub' 2024-04-06T07:41:57.867 INFO:teuthology.orchestra.run.smithi129.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T07:41:57.895 INFO:teuthology.orchestra.run.smithi129.stdout:Generating grub configuration file ... 2024-04-06T07:41:58.168 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T07:41:58.186 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T07:41:58.202 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-04-06T07:41:58.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T07:41:58.284 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-04-06T07:41:58.454 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T07:41:58.460 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T07:41:58.482 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-04-06T07:41:58.482 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-04-06T07:41:58.528 INFO:teuthology.orchestra.run.smithi129.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T07:41:58.533 INFO:teuthology.orchestra.run.smithi129.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T07:41:58.663 INFO:teuthology.orchestra.run.smithi047.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-06T07:41:58.663 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T07:41:58.664 INFO:teuthology.orchestra.run.smithi047.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T07:41:58.664 INFO:teuthology.orchestra.run.smithi047.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T07:41:58.664 INFO:teuthology.orchestra.run.smithi047.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T07:41:58.664 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T07:41:58.664 INFO:teuthology.orchestra.run.smithi047.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T07:41:58.664 INFO:teuthology.orchestra.run.smithi047.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T07:41:58.706 INFO:teuthology.orchestra.run.smithi047.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T07:41:58.708 DEBUG:teuthology.orchestra.run.smithi047:> dpkg -s linux-image-generic 2024-04-06T07:41:58.716 INFO:teuthology.orchestra.run.smithi129.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T07:41:58.716 INFO:teuthology.orchestra.run.smithi129.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T07:41:58.716 INFO:teuthology.orchestra.run.smithi129.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Package: linux-image-generic 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Status: install ok installed 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Priority: optional 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Section: kernel 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Installed-Size: 21 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: amd64 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Source: linux-meta 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Version: 5.15.0.101.98 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T07:41:58.724 INFO:teuthology.orchestra.run.smithi047.stdout:Recommends: thermald 2024-04-06T07:41:58.725 INFO:teuthology.orchestra.run.smithi047.stdout:Description: Generic Linux kernel image 2024-04-06T07:41:58.725 INFO:teuthology.orchestra.run.smithi047.stdout: This package will always depend on the latest generic kernel image 2024-04-06T07:41:58.725 INFO:teuthology.orchestra.run.smithi047.stdout: available. 2024-04-06T07:41:58.725 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-04-06T07:41:58.756 INFO:teuthology.orchestra.run.smithi129.stdout:done 2024-04-06T07:41:58.772 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.NDQWY4uNbo 2024-04-06T07:41:58.772 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NDQWY4uNbo 2024-04-06T07:41:58.828 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.NDQWY4uNbo 2024-04-06T07:41:58.953 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.NDQWY4uNbo is 10KB 2024-04-06T07:41:58.966 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.NDQWY4uNbo 2024-04-06T07:41:58.971 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-06T07:41:59.028 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:41:59.028 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-06T07:41:59.028 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-06T07:41:59.095 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-06T07:41:59.096 DEBUG:teuthology.orchestra.run.smithi047:> sudo update-grub 2024-04-06T07:41:59.230 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2024-04-06T07:41:59.252 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2024-04-06T07:41:59.252 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2024-04-06T07:41:59.252 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2024-04-06T07:41:59.252 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 21 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.15.0.101.98 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2024-04-06T07:41:59.253 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2024-04-06T07:41:59.254 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2024-04-06T07:41:59.254 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2024-04-06T07:41:59.254 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-06T07:41:59.254 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T07:41:59.254 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T07:41:59.254 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-04-06T07:41:59.254 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T07:41:59.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get clean 2024-04-06T07:41:59.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-04-06T07:41:59.585 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T07:41:59.585 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T07:41:59.593 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T07:41:59.613 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T07:42:00.256 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub' 2024-04-06T07:42:00.258 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T07:42:00.260 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-04-06T07:42:00.457 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T07:42:00.463 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T07:42:00.680 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T07:42:00.686 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T07:42:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T07:42:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T07:42:00.869 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T07:42:00.869 INFO:teuthology.orchestra.run.smithi047.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T07:42:00.869 INFO:teuthology.orchestra.run.smithi047.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T07:42:00.883 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-04-06T07:42:00.891 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-04-06T07:42:00.993 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-04-06T07:42:01.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T07:42:01.070 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-04-06T07:42:01.275 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-04-06T07:42:01.276 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-04-06T07:42:01.442 INFO:teuthology.orchestra.run.smithi129.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-06T07:42:01.442 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T07:42:01.442 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T07:42:01.442 INFO:teuthology.orchestra.run.smithi129.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T07:42:01.442 INFO:teuthology.orchestra.run.smithi129.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T07:42:01.442 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T07:42:01.443 INFO:teuthology.orchestra.run.smithi129.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T07:42:01.443 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T07:42:01.480 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T07:42:01.482 DEBUG:teuthology.orchestra.run.smithi129:> dpkg -s linux-image-generic 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Package: linux-image-generic 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Status: install ok installed 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Priority: optional 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Section: kernel 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Installed-Size: 21 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: amd64 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Source: linux-meta 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Version: 5.15.0.101.98 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Recommends: thermald 2024-04-06T07:42:01.495 INFO:teuthology.orchestra.run.smithi129.stdout:Description: Generic Linux kernel image 2024-04-06T07:42:01.496 INFO:teuthology.orchestra.run.smithi129.stdout: This package will always depend on the latest generic kernel image 2024-04-06T07:42:01.496 INFO:teuthology.orchestra.run.smithi129.stdout: available. 2024-04-06T07:42:01.496 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-04-06T07:42:01.541 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.79qOiRRx5y 2024-04-06T07:42:01.541 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub/grub.cfg /tmp/tmp.79qOiRRx5y 2024-04-06T07:42:01.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.79qOiRRx5y 2024-04-06T07:42:01.713 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.79qOiRRx5y is 10KB 2024-04-06T07:42:01.724 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.79qOiRRx5y 2024-04-06T07:42:01.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-06T07:42:01.784 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:42:01.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-06T07:42:01.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-06T07:42:01.847 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-06T07:42:01.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo update-grub 2024-04-06T07:42:03.033 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub' 2024-04-06T07:42:03.034 INFO:teuthology.orchestra.run.smithi129.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T07:42:03.036 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-04-06T07:42:03.203 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T07:42:03.209 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T07:42:03.474 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T07:42:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T07:42:03.521 INFO:teuthology.orchestra.run.smithi129.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T07:42:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T07:42:03.664 INFO:teuthology.orchestra.run.smithi129.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T07:42:03.664 INFO:teuthology.orchestra.run.smithi129.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T07:42:03.664 INFO:teuthology.orchestra.run.smithi129.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T07:42:03.674 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-04-06T07:42:03.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-04-06T07:42:30.920 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-04-06T07:42:30.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-06T07:42:30.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:42:33.682 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-04-06T07:42:33.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-06T07:42:33.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:43:08.319 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-04-06T07:43:17.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-06T07:43:17.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:43:30.964 DEBUG:teuthology.orchestra.remote:timed out 2024-04-06T07:43:35.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-04-06T07:43:39.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-06T07:43:39.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:43:39.977 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-04-06T07:43:47.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-06T07:43:47.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:43:48.306 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-04-06T07:43:49.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-06T07:43:49.118 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-101-generic"... 2024-04-06T07:43:49.118 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-04-06T07:43:49.165 INFO:teuthology.orchestra.run.smithi129.stdout:5.15.0-101-generic 2024-04-06T07:43:49.165 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-06T07:43:49.165 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T07:43:49.165 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-06T07:43:50.166 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-06T07:43:50.166 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T07:43:50.260 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-04-06T07:43:50.291 DEBUG:teuthology.parallel:result is None 2024-04-06T07:43:51.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-06T07:43:51.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:43:52.209 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-04-06T07:43:52.798 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-06T07:43:52.798 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-101-generic"... 2024-04-06T07:43:52.798 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-04-06T07:43:52.846 INFO:teuthology.orchestra.run.smithi047.stdout:5.15.0-101-generic 2024-04-06T07:43:52.846 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-06T07:43:52.846 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T07:43:52.847 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-06T07:43:53.847 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-06T07:43:53.848 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T07:43:53.918 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-04-06T07:43:53.956 DEBUG:teuthology.parallel:result is None 2024-04-06T07:43:53.956 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T07:43:53.963 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T07:43:53.963 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T07:43:53.966 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T07:43:53.972 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T07:43:53.978 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T07:43:53.989 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T07:43:53.990 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T07:43:54.016 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T07:43:54.038 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T07:43:54.045 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T07:43:54.045 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T07:43:54.067 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T07:43:54.083 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T07:43:54.092 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T07:43:54.130 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T07:43:54.138 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T07:43:54.139 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T07:43:54.146 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T07:43:54.147 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T07:43:54.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T07:43:54.162 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T07:43:54.171 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T07:43:54.172 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T07:43:54.203 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T07:43:54.209 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T07:43:54.255 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T07:43:54.303 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:43:54.303 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T07:43:54.364 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T07:43:54.369 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T07:43:54.418 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:43:54.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T07:43:54.481 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-04-06T07:43:54.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-04-06T07:43:54.558 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T07:43:54.569 INFO:teuthology.task.internal:Starting timer... 2024-04-06T07:43:54.569 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T07:43:54.581 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T07:43:54.598 DEBUG:teuthology.task.selinux:Excluding smithi047: OS 'ubuntu' does not support SELinux 2024-04-06T07:43:54.598 DEBUG:teuthology.task.selinux:Excluding smithi129: OS 'ubuntu' does not support SELinux 2024-04-06T07:43:54.599 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T07:43:54.599 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-06T07:43:54.599 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T07:43:54.599 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T07:43:54.608 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-06T07:43:54.614 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-06T07:43:54.749 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T07:43:54.768 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T07:43:54.770 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T07:50:59.930 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-04-06T07:50:59.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-06T07:50:59.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:51:00.028 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-04-06T07:51:00.106 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-04-06T07:51:00.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-06T07:51:00.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T07:51:00.195 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-04-06T07:51:00.270 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-06T07:51:00.270 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T07:51:00.284 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T07:51:00.284 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T07:51:00.285 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T07:51:00.287 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T07:51:00.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T07:51:00.315 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Command line: ntpd -gq 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: ---------------------------------------------------- 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: ntp-4 is maintained by Network Time Foundation, 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: corporation. Support and training for ntp-4 are 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: available at https://www.nwtime.org/support 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: ---------------------------------------------------- 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: proto: precision = 0.066 usec (-24) 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: basedate set to 2022-02-04 2024-04-06T07:51:00.316 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: gps base set to 2022-02-06 (week 2196) 2024-04-06T07:51:00.317 INFO:teuthology.orchestra.run.smithi047.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T07:51:00.317 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T07:51:00.317 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: restrict ::: KOD does nothing without LIMITED. 2024-04-06T07:51:00.317 INFO:teuthology.orchestra.run.smithi047.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-06T07:51:00.318 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listen and drop on 0 v6wildcard [::]:123 2024-04-06T07:51:00.318 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-06T07:51:00.318 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listen normally on 2 lo 127.0.0.1:123 2024-04-06T07:51:00.319 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listen normally on 3 ens1f0 172.21.15.47:123 2024-04-06T07:51:00.319 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listen normally on 4 lo [::1]:123 2024-04-06T07:51:00.319 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:148e%4]:123 2024-04-06T07:51:00.319 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:00 ntpd[17072]: Listening on routing socket on fd #22 for interface updates 2024-04-06T07:51:00.336 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-06T07:51:00.336 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Command line: ntpd -gq 2024-04-06T07:51:00.336 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: ---------------------------------------------------- 2024-04-06T07:51:00.336 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: ntp-4 is maintained by Network Time Foundation, 2024-04-06T07:51:00.336 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-06T07:51:00.337 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: corporation. Support and training for ntp-4 are 2024-04-06T07:51:00.337 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: available at https://www.nwtime.org/support 2024-04-06T07:51:00.337 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: ---------------------------------------------------- 2024-04-06T07:51:00.337 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: proto: precision = 0.043 usec (-24) 2024-04-06T07:51:00.338 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: basedate set to 2022-02-04 2024-04-06T07:51:00.338 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: gps base set to 2022-02-06 (week 2196) 2024-04-06T07:51:00.338 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T07:51:00.338 INFO:teuthology.orchestra.run.smithi129.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T07:51:00.338 INFO:teuthology.orchestra.run.smithi129.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: restrict ::: KOD does nothing without LIMITED. 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listen and drop on 0 v6wildcard [::]:123 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listen normally on 2 lo 127.0.0.1:123 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listen normally on 3 enp3s0f1 172.21.15.129:123 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listen normally on 4 lo [::1]:123 2024-04-06T07:51:00.339 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7261%5]:123 2024-04-06T07:51:00.340 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:00 ntpd[17186]: Listening on routing socket on fd #22 for interface updates 2024-04-06T07:51:01.315 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:01 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:01.315 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:01 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:01.336 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:01 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:01.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:01 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:02.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:02.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:02.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:02 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:02.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:02.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:02.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:02.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:02.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:02 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:04.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:04.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:04 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:04.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:04.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:04 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:05.315 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:05 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:05.315 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:05 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:05.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:05 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:05.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:05 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:05.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:05 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:05.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:05 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:05.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:05 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:05.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:05 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:05.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:05 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:05.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:05 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:05.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:05 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:05.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:05 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:06.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:06.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:06.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:06 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:06.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:06.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:06 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:07.315 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:07 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:07.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:07 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:07.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:07 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:07.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:07 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:07.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:07 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:07.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:07 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:07.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:07 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:07.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:07 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:07.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:07 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:07.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:07 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:07.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:07 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:07.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:07 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:08.316 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:08.317 INFO:teuthology.orchestra.run.smithi047.stdout: 6 Apr 07:51:08 ntpd[17072]: ntpd: time slew -0.003105 s 2024-04-06T07:51:08.317 INFO:teuthology.orchestra.run.smithi047.stdout:ntpd: time slew -0.003105s 2024-04-06T07:51:08.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-06T07:51:08.317 INFO:teuthology.orchestra.run.smithi047.stderr: 6 Apr 07:51:08 ntpd[17072]: can't open /var/log/ntpstats/loopstats.20240406: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T07:51:08.337 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T07:51:08.338 INFO:teuthology.orchestra.run.smithi129.stdout: 6 Apr 07:51:08 ntpd[17186]: ntpd: time slew -0.001481 s 2024-04-06T07:51:08.338 INFO:teuthology.orchestra.run.smithi129.stdout:ntpd: time slew -0.001481s 2024-04-06T07:51:08.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-06T07:51:08.338 INFO:teuthology.orchestra.run.smithi129.stderr: 6 Apr 07:51:08 ntpd[17186]: can't open /var/log/ntpstats/loopstats.20240406: Permission denied 2024-04-06T07:51:08.372 INFO:teuthology.orchestra.run.smithi047.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T07:51:08.372 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================== 2024-04-06T07:51:08.373 INFO:teuthology.orchestra.run.smithi047.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.373 INFO:teuthology.orchestra.run.smithi047.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.373 INFO:teuthology.orchestra.run.smithi047.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.373 INFO:teuthology.orchestra.run.smithi047.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.382 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T07:51:08.382 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2024-04-06T07:51:08.382 INFO:teuthology.orchestra.run.smithi129.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.382 INFO:teuthology.orchestra.run.smithi129.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.382 INFO:teuthology.orchestra.run.smithi129.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.382 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T07:51:08.383 INFO:teuthology.run_tasks:Running task install... 2024-04-06T07:51:08.393 DEBUG:teuthology.task.install:project ceph 2024-04-06T07:51:08.393 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}} 2024-04-06T07:51:08.393 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'} 2024-04-06T07:51:08.394 INFO:teuthology.task.install:Using flavor: default 2024-04-06T07:51:08.399 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-06T07:51:08.399 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-04-06T07:51:08.400 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-key list | grep Ceph 2024-04-06T07:51:08.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-key list | grep Ceph 2024-04-06T07:51:08.473 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-06T07:51:08.486 INFO:teuthology.orchestra.run.smithi129.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-06T07:51:08.502 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-06T07:51:08.502 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph.com (release key) 2024-04-06T07:51:08.503 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-04-06T07:51:08.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T07:51:08.518 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-06T07:51:08.518 INFO:teuthology.orchestra.run.smithi129.stdout:uid [ unknown] Ceph.com (release key) 2024-04-06T07:51:08.519 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-04-06T07:51:08.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T07:51:08.665 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default/ 2024-04-06T07:51:08.665 INFO:teuthology.task.install.deb:Package version is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:51:08.740 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default/ 2024-04-06T07:51:08.740 INFO:teuthology.task.install.deb:Package version is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:51:08.847 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:51:08.848 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-06T07:51:08.863 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-04-06T07:51:08.920 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:51:08.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-06T07:51:08.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo apt-get update 2024-04-06T07:51:09.037 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T07:51:09.037 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T07:51:09.052 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T07:51:09.072 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T07:51:09.113 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T07:51:09.113 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T07:51:09.126 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T07:51:09.149 INFO:teuthology.orchestra.run.smithi129.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T07:51:09.150 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-06T07:51:09.186 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-06T07:51:09.222 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-06T07:51:09.223 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-06T07:51:09.258 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-06T07:51:09.259 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2024-04-06T07:51:09.296 INFO:teuthology.orchestra.run.smithi129.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-06T07:51:09.304 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-06T07:51:09.332 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-06T07:51:09.376 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2024-04-06T07:51:09.414 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 28.5 kB in 0s (78.1 kB/s) 2024-04-06T07:51:09.475 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 28.5 kB in 0s (77.6 kB/s) 2024-04-06T07:51:10.361 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-04-06T07:51:10.378 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1059-g4d05f4ca-1jammy cephadm=18.2.2-1059-g4d05f4ca-1jammy ceph-mds=18.2.2-1059-g4d05f4ca-1jammy ceph-mgr=18.2.2-1059-g4d05f4ca-1jammy ceph-common=18.2.2-1059-g4d05f4ca-1jammy ceph-fuse=18.2.2-1059-g4d05f4ca-1jammy ceph-test=18.2.2-1059-g4d05f4ca-1jammy ceph-volume=18.2.2-1059-g4d05f4ca-1jammy radosgw=18.2.2-1059-g4d05f4ca-1jammy python3-rados=18.2.2-1059-g4d05f4ca-1jammy python3-rgw=18.2.2-1059-g4d05f4ca-1jammy python3-cephfs=18.2.2-1059-g4d05f4ca-1jammy python3-rbd=18.2.2-1059-g4d05f4ca-1jammy libcephfs2=18.2.2-1059-g4d05f4ca-1jammy libcephfs-dev=18.2.2-1059-g4d05f4ca-1jammy librados2=18.2.2-1059-g4d05f4ca-1jammy librbd1=18.2.2-1059-g4d05f4ca-1jammy rbd-fuse=18.2.2-1059-g4d05f4ca-1jammy rbd-mirror=18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:51:10.416 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-04-06T07:51:10.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1059-g4d05f4ca-1jammy cephadm=18.2.2-1059-g4d05f4ca-1jammy ceph-mds=18.2.2-1059-g4d05f4ca-1jammy ceph-mgr=18.2.2-1059-g4d05f4ca-1jammy ceph-common=18.2.2-1059-g4d05f4ca-1jammy ceph-fuse=18.2.2-1059-g4d05f4ca-1jammy ceph-test=18.2.2-1059-g4d05f4ca-1jammy ceph-volume=18.2.2-1059-g4d05f4ca-1jammy radosgw=18.2.2-1059-g4d05f4ca-1jammy python3-rados=18.2.2-1059-g4d05f4ca-1jammy python3-rgw=18.2.2-1059-g4d05f4ca-1jammy python3-cephfs=18.2.2-1059-g4d05f4ca-1jammy python3-rbd=18.2.2-1059-g4d05f4ca-1jammy libcephfs2=18.2.2-1059-g4d05f4ca-1jammy libcephfs-dev=18.2.2-1059-g4d05f4ca-1jammy librados2=18.2.2-1059-g4d05f4ca-1jammy librbd1=18.2.2-1059-g4d05f4ca-1jammy rbd-fuse=18.2.2-1059-g4d05f4ca-1jammy rbd-mirror=18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:51:10.445 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-04-06T07:51:10.492 INFO:teuthology.orchestra.run.smithi129.stdout:Reading package lists... 2024-04-06T07:51:10.646 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-04-06T07:51:10.646 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-04-06T07:51:10.689 INFO:teuthology.orchestra.run.smithi129.stdout:Building dependency tree... 2024-04-06T07:51:10.689 INFO:teuthology.orchestra.run.smithi129.stdout:Reading state information... 2024-04-06T07:51:10.848 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T07:51:10.848 INFO:teuthology.orchestra.run.smithi047.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T07:51:10.848 INFO:teuthology.orchestra.run.smithi047.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T07:51:10.848 INFO:teuthology.orchestra.run.smithi047.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T07:51:10.849 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T07:51:10.849 INFO:teuthology.orchestra.run.smithi047.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T07:51:10.849 INFO:teuthology.orchestra.run.smithi047.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T07:51:10.849 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2024-04-06T07:51:10.850 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-06T07:51:10.850 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-06T07:51:10.850 INFO:teuthology.orchestra.run.smithi047.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-04-06T07:51:10.850 INFO:teuthology.orchestra.run.smithi047.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-06T07:51:10.851 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-04-06T07:51:10.851 INFO:teuthology.orchestra.run.smithi047.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-06T07:51:10.851 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-06T07:51:10.851 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T07:51:10.852 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: python3-watchdog 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout:Recommended packages: 2024-04-06T07:51:10.853 INFO:teuthology.orchestra.run.smithi047.stdout: btrfs-tools 2024-04-06T07:51:10.871 INFO:teuthology.orchestra.run.smithi129.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T07:51:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T07:51:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T07:51:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T07:51:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T07:51:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T07:51:10.872 INFO:teuthology.orchestra.run.smithi129.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T07:51:10.873 INFO:teuthology.orchestra.run.smithi129.stdout:The following additional packages will be installed: 2024-04-06T07:51:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T07:51:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout:Suggested packages: 2024-04-06T07:51:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: python3-watchdog 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout:Recommended packages: 2024-04-06T07:51:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: btrfs-tools 2024-04-06T07:51:10.960 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2024-04-06T07:51:10.960 INFO:teuthology.orchestra.run.smithi047.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-06T07:51:10.960 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-06T07:51:10.960 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-06T07:51:10.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-06T07:51:10.963 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror socat xmlstarlet 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout:The following NEW packages will be installed: 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-04-06T07:51:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-06T07:51:10.977 INFO:teuthology.orchestra.run.smithi129.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-06T07:51:10.978 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror socat xmlstarlet 2024-04-06T07:51:11.043 INFO:teuthology.orchestra.run.smithi129.stdout:0 upgraded, 110 newly installed, 0 to remove and 299 not upgraded. 2024-04-06T07:51:11.043 INFO:teuthology.orchestra.run.smithi129.stdout:Need to get 165 MB of archives. 2024-04-06T07:51:11.044 INFO:teuthology.orchestra.run.smithi129.stdout:After this operation, 673 MB of additional disk space will be used. 2024-04-06T07:51:11.044 INFO:teuthology.orchestra.run.smithi129.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-04-06T07:51:11.113 INFO:teuthology.orchestra.run.smithi129.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-06T07:51:11.118 INFO:teuthology.orchestra.run.smithi129.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,440 kB] 2024-04-06T07:51:11.151 INFO:teuthology.orchestra.run.smithi047.stdout:0 upgraded, 110 newly installed, 0 to remove and 299 not upgraded. 2024-04-06T07:51:11.151 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 165 MB of archives. 2024-04-06T07:51:11.151 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 673 MB of additional disk space will be used. 2024-04-06T07:51:11.151 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-04-06T07:51:11.157 INFO:teuthology.orchestra.run.smithi129.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-06T07:51:11.160 INFO:teuthology.orchestra.run.smithi129.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-06T07:51:11.181 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,440 kB] 2024-04-06T07:51:11.182 INFO:teuthology.orchestra.run.smithi129.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-06T07:51:11.249 INFO:teuthology.orchestra.run.smithi129.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-06T07:51:11.254 INFO:teuthology.orchestra.run.smithi129.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-06T07:51:11.266 INFO:teuthology.orchestra.run.smithi129.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-06T07:51:11.271 INFO:teuthology.orchestra.run.smithi129.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-06T07:51:11.271 INFO:teuthology.orchestra.run.smithi129.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-06T07:51:11.272 INFO:teuthology.orchestra.run.smithi129.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-06T07:51:11.281 INFO:teuthology.orchestra.run.smithi129.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-06T07:51:11.282 INFO:teuthology.orchestra.run.smithi129.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-06T07:51:11.283 INFO:teuthology.orchestra.run.smithi129.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-06T07:51:11.283 INFO:teuthology.orchestra.run.smithi129.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-06T07:51:11.313 INFO:teuthology.orchestra.run.smithi129.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-06T07:51:11.314 INFO:teuthology.orchestra.run.smithi129.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-06T07:51:11.314 INFO:teuthology.orchestra.run.smithi129.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-06T07:51:11.314 INFO:teuthology.orchestra.run.smithi129.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-06T07:51:11.315 INFO:teuthology.orchestra.run.smithi129.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-06T07:51:11.315 INFO:teuthology.orchestra.run.smithi129.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-06T07:51:11.336 INFO:teuthology.orchestra.run.smithi129.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,105 kB] 2024-04-06T07:51:11.355 INFO:teuthology.orchestra.run.smithi129.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-06T07:51:11.356 INFO:teuthology.orchestra.run.smithi129.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-06T07:51:11.357 INFO:teuthology.orchestra.run.smithi129.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-06T07:51:11.358 INFO:teuthology.orchestra.run.smithi129.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-06T07:51:11.401 INFO:teuthology.orchestra.run.smithi129.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-06T07:51:11.401 INFO:teuthology.orchestra.run.smithi129.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-06T07:51:11.402 INFO:teuthology.orchestra.run.smithi129.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-06T07:51:11.405 INFO:teuthology.orchestra.run.smithi129.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-06T07:51:11.406 INFO:teuthology.orchestra.run.smithi129.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-06T07:51:11.407 INFO:teuthology.orchestra.run.smithi129.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-06T07:51:11.426 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-06T07:51:11.437 INFO:teuthology.orchestra.run.smithi129.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1059-g4d05f4ca-1jammy [679 kB] 2024-04-06T07:51:11.449 INFO:teuthology.orchestra.run.smithi129.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-06T07:51:11.449 INFO:teuthology.orchestra.run.smithi129.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-06T07:51:11.451 INFO:teuthology.orchestra.run.smithi129.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-06T07:51:11.452 INFO:teuthology.orchestra.run.smithi129.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-06T07:51:11.459 INFO:teuthology.orchestra.run.smithi129.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1059-g4d05f4ca-1jammy [356 kB] 2024-04-06T07:51:11.474 INFO:teuthology.orchestra.run.smithi129.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1059-g4d05f4ca-1jammy [32.8 kB] 2024-04-06T07:51:11.476 INFO:teuthology.orchestra.run.smithi129.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1059-g4d05f4ca-1jammy [180 kB] 2024-04-06T07:51:11.485 INFO:teuthology.orchestra.run.smithi129.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1059-g4d05f4ca-1jammy [63.7 kB] 2024-04-06T07:51:11.487 INFO:teuthology.orchestra.run.smithi129.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1059-g4d05f4ca-1jammy [330 kB] 2024-04-06T07:51:11.498 INFO:teuthology.orchestra.run.smithi129.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1059-g4d05f4ca-1jammy [5,890 kB] 2024-04-06T07:51:11.499 INFO:teuthology.orchestra.run.smithi129.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-06T07:51:11.503 INFO:teuthology.orchestra.run.smithi129.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-06T07:51:11.505 INFO:teuthology.orchestra.run.smithi129.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-06T07:51:11.508 INFO:teuthology.orchestra.run.smithi129.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-06T07:51:11.509 INFO:teuthology.orchestra.run.smithi129.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-06T07:51:11.514 INFO:teuthology.orchestra.run.smithi129.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-06T07:51:11.554 INFO:teuthology.orchestra.run.smithi129.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-06T07:51:11.559 INFO:teuthology.orchestra.run.smithi129.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-06T07:51:11.598 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-06T07:51:11.612 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-06T07:51:11.684 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-06T07:51:11.707 INFO:teuthology.orchestra.run.smithi129.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1059-g4d05f4ca-1jammy [112 kB] 2024-04-06T07:51:11.710 INFO:teuthology.orchestra.run.smithi129.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1059-g4d05f4ca-1jammy [441 kB] 2024-04-06T07:51:11.727 INFO:teuthology.orchestra.run.smithi129.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1059-g4d05f4ca-1jammy [21.9 MB] 2024-04-06T07:51:11.756 INFO:teuthology.orchestra.run.smithi129.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-06T07:51:11.758 INFO:teuthology.orchestra.run.smithi129.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-06T07:51:11.759 INFO:teuthology.orchestra.run.smithi129.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-06T07:51:11.760 INFO:teuthology.orchestra.run.smithi129.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-06T07:51:11.760 INFO:teuthology.orchestra.run.smithi129.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-06T07:51:11.762 INFO:teuthology.orchestra.run.smithi129.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-06T07:51:11.762 INFO:teuthology.orchestra.run.smithi129.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-06T07:51:11.763 INFO:teuthology.orchestra.run.smithi129.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-06T07:51:11.766 INFO:teuthology.orchestra.run.smithi129.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-06T07:51:11.788 INFO:teuthology.orchestra.run.smithi129.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-06T07:51:11.796 INFO:teuthology.orchestra.run.smithi129.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-06T07:51:11.827 INFO:teuthology.orchestra.run.smithi129.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-06T07:51:11.829 INFO:teuthology.orchestra.run.smithi129.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-06T07:51:11.830 INFO:teuthology.orchestra.run.smithi129.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-06T07:51:11.872 INFO:teuthology.orchestra.run.smithi129.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-06T07:51:11.873 INFO:teuthology.orchestra.run.smithi129.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-06T07:51:11.873 INFO:teuthology.orchestra.run.smithi129.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-06T07:51:11.874 INFO:teuthology.orchestra.run.smithi129.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-06T07:51:11.875 INFO:teuthology.orchestra.run.smithi129.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-06T07:51:11.879 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-06T07:51:11.880 INFO:teuthology.orchestra.run.smithi129.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-06T07:51:11.881 INFO:teuthology.orchestra.run.smithi129.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-06T07:51:11.882 INFO:teuthology.orchestra.run.smithi129.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-06T07:51:11.887 INFO:teuthology.orchestra.run.smithi129.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-06T07:51:11.888 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-06T07:51:11.901 INFO:teuthology.orchestra.run.smithi129.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-06T07:51:11.903 INFO:teuthology.orchestra.run.smithi129.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-06T07:51:11.904 INFO:teuthology.orchestra.run.smithi129.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-06T07:51:11.910 INFO:teuthology.orchestra.run.smithi129.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-06T07:51:11.914 INFO:teuthology.orchestra.run.smithi129.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-06T07:51:11.919 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-06T07:51:11.920 INFO:teuthology.orchestra.run.smithi129.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-06T07:51:11.928 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-06T07:51:11.929 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-06T07:51:11.930 INFO:teuthology.orchestra.run.smithi129.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-06T07:51:11.930 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-06T07:51:11.931 INFO:teuthology.orchestra.run.smithi129.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-06T07:51:11.933 INFO:teuthology.orchestra.run.smithi129.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-06T07:51:11.943 INFO:teuthology.orchestra.run.smithi129.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-06T07:51:11.968 INFO:teuthology.orchestra.run.smithi129.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-06T07:51:11.969 INFO:teuthology.orchestra.run.smithi129.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-06T07:51:12.112 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-06T07:51:12.324 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-06T07:51:12.338 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-06T07:51:12.456 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-06T07:51:12.464 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-06T07:51:12.475 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-06T07:51:12.491 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-06T07:51:12.505 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-06T07:51:12.510 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-06T07:51:12.514 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-06T07:51:12.531 INFO:teuthology.orchestra.run.smithi129.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1059-g4d05f4ca-1jammy [5,001 kB] 2024-04-06T07:51:12.617 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-06T07:51:12.673 INFO:teuthology.orchestra.run.smithi129.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1059-g4d05f4ca-1jammy [241 kB] 2024-04-06T07:51:12.680 INFO:teuthology.orchestra.run.smithi129.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [130 kB] 2024-04-06T07:51:12.684 INFO:teuthology.orchestra.run.smithi129.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1059-g4d05f4ca-1jammy [1,490 kB] 2024-04-06T07:51:12.738 INFO:teuthology.orchestra.run.smithi129.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1059-g4d05f4ca-1jammy [6,079 kB] 2024-04-06T07:51:12.792 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-06T07:51:12.896 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1059-g4d05f4ca-1jammy [3,105 kB] 2024-04-06T07:51:12.945 INFO:teuthology.orchestra.run.smithi129.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1059-g4d05f4ca-1jammy [22.3 MB] 2024-04-06T07:51:12.959 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-06T07:51:12.983 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-06T07:51:13.113 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-06T07:51:13.133 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-06T07:51:13.140 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-06T07:51:13.211 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-06T07:51:13.239 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-06T07:51:13.246 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-06T07:51:13.369 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-06T07:51:13.370 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-06T07:51:13.538 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-06T07:51:13.728 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-06T07:51:13.831 INFO:teuthology.orchestra.run.smithi129.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [14.1 kB] 2024-04-06T07:51:13.831 INFO:teuthology.orchestra.run.smithi129.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [831 kB] 2024-04-06T07:51:13.847 INFO:teuthology.orchestra.run.smithi129.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1059-g4d05f4ca-1jammy [2,153 kB] 2024-04-06T07:51:13.882 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-06T07:51:13.885 INFO:teuthology.orchestra.run.smithi129.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1059-g4d05f4ca-1jammy [238 kB] 2024-04-06T07:51:13.888 INFO:teuthology.orchestra.run.smithi129.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1059-g4d05f4ca-1jammy [144 kB] 2024-04-06T07:51:13.891 INFO:teuthology.orchestra.run.smithi129.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1059-g4d05f4ca-1jammy [1,932 kB] 2024-04-06T07:51:13.915 INFO:teuthology.orchestra.run.smithi129.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1059-g4d05f4ca-1jammy [8,625 kB] 2024-04-06T07:51:13.988 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-06T07:51:14.010 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-06T07:51:14.042 INFO:teuthology.orchestra.run.smithi047.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-06T07:51:14.055 INFO:teuthology.orchestra.run.smithi047.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-06T07:51:14.095 INFO:teuthology.orchestra.run.smithi047.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-06T07:51:14.103 INFO:teuthology.orchestra.run.smithi047.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-06T07:51:14.133 INFO:teuthology.orchestra.run.smithi047.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-06T07:51:14.220 INFO:teuthology.orchestra.run.smithi129.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1059-g4d05f4ca-1jammy [14.3 kB] 2024-04-06T07:51:14.220 INFO:teuthology.orchestra.run.smithi129.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1059-g4d05f4ca-1jammy [42.6 MB] 2024-04-06T07:51:14.363 INFO:teuthology.orchestra.run.smithi047.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-06T07:51:14.407 INFO:teuthology.orchestra.run.smithi047.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1059-g4d05f4ca-1jammy [679 kB] 2024-04-06T07:51:14.529 INFO:teuthology.orchestra.run.smithi047.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-06T07:51:14.755 INFO:teuthology.orchestra.run.smithi047.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1059-g4d05f4ca-1jammy [356 kB] 2024-04-06T07:51:14.887 INFO:teuthology.orchestra.run.smithi047.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-06T07:51:14.904 INFO:teuthology.orchestra.run.smithi047.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-06T07:51:14.916 INFO:teuthology.orchestra.run.smithi047.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-06T07:51:14.938 INFO:teuthology.orchestra.run.smithi047.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1059-g4d05f4ca-1jammy [32.8 kB] 2024-04-06T07:51:14.958 INFO:teuthology.orchestra.run.smithi047.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1059-g4d05f4ca-1jammy [180 kB] 2024-04-06T07:51:14.969 INFO:teuthology.orchestra.run.smithi047.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-06T07:51:14.973 INFO:teuthology.orchestra.run.smithi047.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-06T07:51:14.982 INFO:teuthology.orchestra.run.smithi047.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-06T07:51:15.050 INFO:teuthology.orchestra.run.smithi047.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1059-g4d05f4ca-1jammy [63.7 kB] 2024-04-06T07:51:15.069 INFO:teuthology.orchestra.run.smithi047.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-06T07:51:15.072 INFO:teuthology.orchestra.run.smithi047.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-06T07:51:15.078 INFO:teuthology.orchestra.run.smithi047.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1059-g4d05f4ca-1jammy [330 kB] 2024-04-06T07:51:15.093 INFO:teuthology.orchestra.run.smithi047.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-06T07:51:15.243 INFO:teuthology.orchestra.run.smithi047.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1059-g4d05f4ca-1jammy [5,890 kB] 2024-04-06T07:51:15.286 INFO:teuthology.orchestra.run.smithi047.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-06T07:51:15.458 INFO:teuthology.orchestra.run.smithi047.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-06T07:51:15.631 INFO:teuthology.orchestra.run.smithi047.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-06T07:51:15.982 INFO:teuthology.orchestra.run.smithi129.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1059-g4d05f4ca-1jammy [121 kB] 2024-04-06T07:51:15.984 INFO:teuthology.orchestra.run.smithi129.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1059-g4d05f4ca-1jammy [38.3 kB] 2024-04-06T07:51:15.985 INFO:teuthology.orchestra.run.smithi129.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1059-g4d05f4ca-1jammy [11.6 MB] 2024-04-06T07:51:16.138 INFO:teuthology.orchestra.run.smithi047.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-06T07:51:16.138 INFO:teuthology.orchestra.run.smithi047.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-06T07:51:16.141 INFO:teuthology.orchestra.run.smithi047.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-06T07:51:16.143 INFO:teuthology.orchestra.run.smithi047.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-06T07:51:16.145 INFO:teuthology.orchestra.run.smithi047.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-06T07:51:16.147 INFO:teuthology.orchestra.run.smithi047.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-06T07:51:16.148 INFO:teuthology.orchestra.run.smithi047.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-06T07:51:16.150 INFO:teuthology.orchestra.run.smithi047.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-06T07:51:16.168 INFO:teuthology.orchestra.run.smithi047.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-06T07:51:16.271 INFO:teuthology.orchestra.run.smithi129.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [93.1 kB] 2024-04-06T07:51:16.273 INFO:teuthology.orchestra.run.smithi129.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.2.2-1059-g4d05f4ca-1jammy [3,031 kB] 2024-04-06T07:51:16.340 INFO:teuthology.orchestra.run.smithi047.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-06T07:51:16.720 INFO:teuthology.orchestra.run.smithi047.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-06T07:51:16.743 INFO:teuthology.orchestra.run.smithi047.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-06T07:51:16.895 INFO:teuthology.orchestra.run.smithi047.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-06T07:51:16.929 INFO:teuthology.orchestra.run.smithi129.stdout:Fetched 165 MB in 5s (30.6 MB/s) 2024-04-06T07:51:16.945 INFO:teuthology.orchestra.run.smithi047.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-06T07:51:16.976 INFO:teuthology.orchestra.run.smithi047.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1059-g4d05f4ca-1jammy [112 kB] 2024-04-06T07:51:17.005 INFO:teuthology.orchestra.run.smithi047.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1059-g4d05f4ca-1jammy [441 kB] 2024-04-06T07:51:17.006 INFO:teuthology.orchestra.run.smithi047.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-06T07:51:17.033 INFO:teuthology.orchestra.run.smithi047.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-06T07:51:17.037 INFO:teuthology.orchestra.run.smithi047.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-06T07:51:17.043 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libfmt8:amd64. 2024-04-06T07:51:17.049 INFO:teuthology.orchestra.run.smithi047.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-06T07:51:17.080 INFO:teuthology.orchestra.run.smithi047.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1059-g4d05f4ca-1jammy [21.9 MB] 2024-04-06T07:51:17.092 INFO:teuthology.orchestra.run.smithi047.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-06T07:51:17.093 INFO:teuthology.orchestra.run.smithi047.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-06T07:51:17.107 INFO:teuthology.orchestra.run.smithi129.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-06T07:51:17.112 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-04-06T07:51:17.193 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T07:51:17.259 INFO:teuthology.orchestra.run.smithi047.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-06T07:51:17.538 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-06T07:51:17.564 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-06T07:51:17.629 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T07:51:17.974 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-06T07:51:18.001 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-06T07:51:18.041 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T07:51:18.311 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-06T07:51:18.337 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-06T07:51:18.369 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T07:51:18.765 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-06T07:51:18.792 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T07:51:18.831 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:51:19.218 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-06T07:51:19.237 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T07:51:19.276 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:51:19.630 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-06T07:51:19.656 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T07:51:19.696 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:51:20.016 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-06T07:51:20.043 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-06T07:51:20.074 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T07:51:20.385 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librados2. 2024-04-06T07:51:20.412 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../008-librados2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:20.452 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:20.636 INFO:teuthology.orchestra.run.smithi047.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1059-g4d05f4ca-1jammy [5,001 kB] 2024-04-06T07:51:20.880 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librbd1. 2024-04-06T07:51:20.906 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../009-librbd1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:20.946 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:21.261 INFO:teuthology.orchestra.run.smithi047.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1059-g4d05f4ca-1jammy [241 kB] 2024-04-06T07:51:21.295 INFO:teuthology.orchestra.run.smithi047.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [130 kB] 2024-04-06T07:51:21.301 INFO:teuthology.orchestra.run.smithi047.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1059-g4d05f4ca-1jammy [1,490 kB] 2024-04-06T07:51:21.400 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs2. 2024-04-06T07:51:21.426 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:21.466 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:21.507 INFO:teuthology.orchestra.run.smithi047.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1059-g4d05f4ca-1jammy [6,079 kB] 2024-04-06T07:51:21.752 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rados. 2024-04-06T07:51:21.778 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:21.819 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:22.097 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-06T07:51:22.112 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:22.155 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:22.260 INFO:teuthology.orchestra.run.smithi047.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1059-g4d05f4ca-1jammy [22.3 MB] 2024-04-06T07:51:22.408 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cephfs. 2024-04-06T07:51:22.423 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:22.466 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:22.785 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-06T07:51:22.812 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:22.852 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:23.189 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-06T07:51:23.215 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-06T07:51:23.255 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T07:51:23.792 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-prettytable. 2024-04-06T07:51:23.818 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-04-06T07:51:23.859 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-06T07:51:24.120 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rbd. 2024-04-06T07:51:24.146 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:24.186 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:24.490 INFO:teuthology.orchestra.run.smithi047.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1059-g4d05f4ca-1jammy [14.1 kB] 2024-04-06T07:51:24.491 INFO:teuthology.orchestra.run.smithi047.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [831 kB] 2024-04-06T07:51:24.507 INFO:teuthology.orchestra.run.smithi047.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1059-g4d05f4ca-1jammy [2,153 kB] 2024-04-06T07:51:24.531 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-06T07:51:24.558 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-06T07:51:24.598 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T07:51:24.643 INFO:teuthology.orchestra.run.smithi047.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1059-g4d05f4ca-1jammy [238 kB] 2024-04-06T07:51:24.674 INFO:teuthology.orchestra.run.smithi047.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1059-g4d05f4ca-1jammy [144 kB] 2024-04-06T07:51:24.677 INFO:teuthology.orchestra.run.smithi047.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1059-g4d05f4ca-1jammy [1,932 kB] 2024-04-06T07:51:24.820 INFO:teuthology.orchestra.run.smithi047.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1059-g4d05f4ca-1jammy [8,625 kB] 2024-04-06T07:51:24.917 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package librgw2. 2024-04-06T07:51:24.944 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../019-librgw2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:24.975 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:25.364 INFO:teuthology.orchestra.run.smithi047.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1059-g4d05f4ca-1jammy [14.3 kB] 2024-04-06T07:51:25.364 INFO:teuthology.orchestra.run.smithi047.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1059-g4d05f4ca-1jammy [42.6 MB] 2024-04-06T07:51:25.461 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rgw. 2024-04-06T07:51:25.477 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:25.520 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:25.865 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-06T07:51:25.891 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-06T07:51:25.931 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T07:51:26.242 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libradosstriper1. 2024-04-06T07:51:26.269 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:26.309 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:26.671 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-common. 2024-04-06T07:51:26.698 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:26.737 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:27.576 INFO:teuthology.orchestra.run.smithi047.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1059-g4d05f4ca-1jammy [121 kB] 2024-04-06T07:51:27.578 INFO:teuthology.orchestra.run.smithi047.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1059-g4d05f4ca-1jammy [38.3 kB] 2024-04-06T07:51:27.580 INFO:teuthology.orchestra.run.smithi047.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1059-g4d05f4ca-1jammy [11.6 MB] 2024-04-06T07:51:27.912 INFO:teuthology.orchestra.run.smithi047.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1059-g4d05f4ca-1jammy [93.1 kB] 2024-04-06T07:51:27.914 INFO:teuthology.orchestra.run.smithi047.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/4d05f4cad464de0fcdd11a2856643e533aea34e8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.2.2-1059-g4d05f4ca-1jammy [3,031 kB] 2024-04-06T07:51:27.993 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-base. 2024-04-06T07:51:28.021 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:28.070 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:28.612 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 165 MB in 17s (9,671 kB/s) 2024-04-06T07:51:28.698 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-06T07:51:28.725 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-06T07:51:28.731 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libfmt8:amd64. 2024-04-06T07:51:28.756 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-06T07:51:28.825 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-06T07:51:28.830 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-04-06T07:51:28.913 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T07:51:29.026 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cheroot. 2024-04-06T07:51:29.052 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-06T07:51:29.092 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T07:51:29.208 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-06T07:51:29.234 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-06T07:51:29.266 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T07:51:29.420 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-06T07:51:29.448 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-06T07:51:29.487 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-06T07:51:29.594 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-06T07:51:29.621 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-06T07:51:29.661 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T07:51:29.790 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-06T07:51:29.816 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-06T07:51:29.857 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-06T07:51:29.997 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-06T07:51:30.024 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-06T07:51:30.066 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T07:51:30.110 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-06T07:51:30.137 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-06T07:51:30.168 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-06T07:51:30.454 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-tempora. 2024-04-06T07:51:30.481 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-04-06T07:51:30.525 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-06T07:51:30.529 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-06T07:51:30.552 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T07:51:30.600 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:51:30.832 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-portend. 2024-04-06T07:51:30.859 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-04-06T07:51:30.899 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-06T07:51:31.037 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-06T07:51:31.064 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T07:51:31.103 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:51:31.160 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-06T07:51:31.175 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-06T07:51:31.218 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-06T07:51:31.513 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-06T07:51:31.540 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-06T07:51:31.556 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-06T07:51:31.580 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-06T07:51:31.583 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T07:51:31.622 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:51:31.908 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-natsort. 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-logutils. 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-06T07:51:32.461 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-06T07:51:32.462 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T07:51:32.462 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librados2. 2024-04-06T07:51:32.462 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../008-librados2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:32.462 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:32.640 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-06T07:51:32.666 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-06T07:51:32.706 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-06T07:51:32.814 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librbd1. 2024-04-06T07:51:32.840 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../009-librbd1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:32.881 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:33.009 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-mako. 2024-04-06T07:51:33.036 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-06T07:51:33.075 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T07:51:33.284 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs2. 2024-04-06T07:51:33.310 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:33.342 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:33.395 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-06T07:51:33.422 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-06T07:51:33.462 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-06T07:51:33.594 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rados. 2024-04-06T07:51:33.610 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:33.652 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:33.698 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-06T07:51:33.725 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-06T07:51:33.765 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-06T07:51:33.980 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-06T07:51:34.007 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:34.047 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:34.243 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webob. 2024-04-06T07:51:34.270 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-06T07:51:34.299 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cephfs. 2024-04-06T07:51:34.309 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-06T07:51:34.315 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:34.357 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:34.660 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-06T07:51:34.662 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-waitress. 2024-04-06T07:51:34.686 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:34.689 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-06T07:51:34.726 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:34.732 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T07:51:34.988 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-06T07:51:35.004 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-06T07:51:35.038 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T07:51:35.041 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-tempita. 2024-04-06T07:51:35.068 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-06T07:51:35.107 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T07:51:35.357 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-prettytable. 2024-04-06T07:51:35.368 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-paste. 2024-04-06T07:51:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-06T07:51:35.385 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-04-06T07:51:35.418 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T07:51:35.423 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-06T07:51:35.667 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rbd. 2024-04-06T07:51:35.684 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:35.726 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:35.730 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-06T07:51:35.746 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-06T07:51:35.805 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T07:51:36.066 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-06T07:51:36.079 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-06T07:51:36.082 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-06T07:51:36.106 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-06T07:51:36.116 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-06T07:51:36.145 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T07:51:36.419 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-webtest. 2024-04-06T07:51:36.440 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librgw2. 2024-04-06T07:51:36.446 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-04-06T07:51:36.467 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../019-librgw2_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:36.486 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-06T07:51:36.498 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:36.806 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pecan. 2024-04-06T07:51:36.833 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-06T07:51:36.872 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T07:51:37.008 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rgw. 2024-04-06T07:51:37.026 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:37.067 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:37.244 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-certifi. 2024-04-06T07:51:37.270 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-04-06T07:51:37.310 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-06T07:51:37.403 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-06T07:51:37.430 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-06T07:51:37.470 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T07:51:37.630 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-idna. 2024-04-06T07:51:37.657 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-04-06T07:51:37.696 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-06T07:51:37.781 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libradosstriper1. 2024-04-06T07:51:37.808 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:37.847 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:38.010 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-urllib3. 2024-04-06T07:51:38.036 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-06T07:51:38.076 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T07:51:38.184 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-common. 2024-04-06T07:51:38.211 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:38.250 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:38.371 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-requests. 2024-04-06T07:51:38.398 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-06T07:51:38.437 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T07:51:38.732 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-06T07:51:38.759 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-06T07:51:38.798 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T07:51:39.093 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-06T07:51:39.109 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:39.539 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-base. 2024-04-06T07:51:39.566 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:39.615 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:39.790 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-06T07:51:39.817 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:39.856 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:40.126 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-openssl. 2024-04-06T07:51:40.135 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-06T07:51:40.141 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-04-06T07:51:40.162 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-06T07:51:40.175 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-06T07:51:40.201 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-06T07:51:40.520 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr. 2024-04-06T07:51:40.521 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cheroot. 2024-04-06T07:51:40.550 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-06T07:51:40.550 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:40.587 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:40.587 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T07:51:40.915 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-06T07:51:40.942 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-06T07:51:40.982 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mon. 2024-04-06T07:51:40.982 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-06T07:51:41.008 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:41.048 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:41.251 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-06T07:51:41.278 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-06T07:51:41.309 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-06T07:51:41.619 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-osd. 2024-04-06T07:51:41.631 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-06T07:51:41.646 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:41.658 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-06T07:51:41.677 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:41.697 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-06T07:51:42.000 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tempora. 2024-04-06T07:51:42.027 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-04-06T07:51:42.067 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-06T07:51:42.328 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-portend. 2024-04-06T07:51:42.344 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-04-06T07:51:42.378 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-06T07:51:42.664 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-06T07:51:42.691 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-06T07:51:42.713 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph. 2024-04-06T07:51:42.729 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../059-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:42.730 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-06T07:51:42.771 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:43.033 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-06T07:51:43.060 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-06T07:51:43.099 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-fuse. 2024-04-06T07:51:43.100 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-06T07:51:43.126 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:43.174 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:43.495 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-natsort. 2024-04-06T07:51:43.510 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mds. 2024-04-06T07:51:43.522 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-04-06T07:51:43.526 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:43.561 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:43.561 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-06T07:51:43.873 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-logutils. 2024-04-06T07:51:43.900 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-04-06T07:51:43.913 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package cephadm. 2024-04-06T07:51:43.935 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../062-cephadm_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:43.939 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-06T07:51:43.980 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:44.217 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-06T07:51:44.244 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-06T07:51:44.776 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-06T07:51:44.967 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-06T07:51:44.994 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-06T07:51:45.033 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-06T07:51:45.137 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-mako. 2024-04-06T07:51:45.164 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-06T07:51:45.203 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T07:51:45.353 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-06T07:51:45.369 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T07:51:45.411 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T07:51:45.540 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-06T07:51:45.567 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-06T07:51:45.607 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-06T07:51:45.909 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-06T07:51:45.936 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-06T07:51:45.975 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-06T07:51:46.270 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webob. 2024-04-06T07:51:46.297 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-06T07:51:46.336 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-06T07:51:46.416 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-babel. 2024-04-06T07:51:46.444 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T07:51:46.483 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T07:51:46.656 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-waitress. 2024-04-06T07:51:46.680 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-06T07:51:46.717 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T07:51:46.844 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-jinja2. 2024-04-06T07:51:46.870 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-06T07:51:46.911 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T07:51:46.959 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tempita. 2024-04-06T07:51:46.986 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-06T07:51:47.017 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T07:51:47.247 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-06T07:51:47.274 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:47.314 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:47.320 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-paste. 2024-04-06T07:51:47.347 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-06T07:51:47.386 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T07:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-06T07:51:47.686 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-06T07:51:47.714 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-06T07:51:47.725 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-06T07:51:47.730 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-06T07:51:47.772 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T07:51:48.028 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pluggy. 2024-04-06T07:51:48.044 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-06T07:51:48.075 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-06T07:51:48.078 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-06T07:51:48.102 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-06T07:51:48.142 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-06T07:51:48.348 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-py. 2024-04-06T07:51:48.374 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-04-06T07:51:48.414 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-06T07:51:48.428 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webtest. 2024-04-06T07:51:48.455 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-04-06T07:51:48.486 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-06T07:51:48.743 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-06T07:51:48.770 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-06T07:51:48.772 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pecan. 2024-04-06T07:51:48.800 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-06T07:51:48.809 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-06T07:51:48.839 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T07:51:49.112 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-toml. 2024-04-06T07:51:49.139 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-04-06T07:51:49.179 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-06T07:51:49.193 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-certifi. 2024-04-06T07:51:49.221 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-04-06T07:51:49.260 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-06T07:51:49.482 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pytest. 2024-04-06T07:51:49.509 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-06T07:51:49.548 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T07:51:49.579 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-idna. 2024-04-06T07:51:49.607 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2024-04-06T07:51:49.646 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-06T07:51:49.957 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-urllib3. 2024-04-06T07:51:49.984 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-06T07:51:50.023 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T07:51:50.227 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-06T07:51:50.255 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-04-06T07:51:50.293 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-06T07:51:50.352 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-requests. 2024-04-06T07:51:50.379 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-06T07:51:50.418 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T07:51:50.596 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-routes. 2024-04-06T07:51:50.624 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-06T07:51:50.663 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T07:51:50.704 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-06T07:51:50.721 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-06T07:51:50.762 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T07:51:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-06T07:51:51.026 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:51.049 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-06T07:51:51.065 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:51.065 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:51.107 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:51.469 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-06T07:51:51.485 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-06T07:51:51.503 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-06T07:51:51.519 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T07:51:51.530 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:51.570 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:51.881 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-openssl. 2024-04-06T07:51:51.908 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2024-04-06T07:51:51.964 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-06T07:51:51.990 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-joblib. 2024-04-06T07:51:52.016 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-06T07:51:52.056 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T07:51:52.267 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr. 2024-04-06T07:51:52.294 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../056-ceph-mgr_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:52.333 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:52.393 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-06T07:51:52.420 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-06T07:51:52.460 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-06T07:51:52.678 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mon. 2024-04-06T07:51:52.705 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../057-ceph-mon_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:52.744 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:52.762 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-sklearn. 2024-04-06T07:51:52.789 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-06T07:51:52.829 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T07:51:53.473 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-osd. 2024-04-06T07:51:53.500 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../058-ceph-osd_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:53.539 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:53.570 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-06T07:51:53.587 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:53.628 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:54.565 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-cachetools. 2024-04-06T07:51:54.592 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-04-06T07:51:54.623 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-06T07:51:54.693 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph. 2024-04-06T07:51:54.720 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../059-ceph_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:54.759 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:55.020 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-fuse. 2024-04-06T07:51:55.047 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../060-ceph-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:55.078 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:55.137 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-06T07:51:55.164 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-06T07:51:55.244 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-06T07:51:55.457 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mds. 2024-04-06T07:51:55.484 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../061-ceph-mds_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:55.523 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:55.598 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-06T07:51:55.625 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-06T07:51:55.664 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T07:51:55.901 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephadm. 2024-04-06T07:51:55.928 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../062-cephadm_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:51:55.967 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:55.984 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-rsa. 2024-04-06T07:51:56.011 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-04-06T07:51:56.050 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-06T07:51:56.254 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-06T07:51:56.269 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-06T07:51:56.312 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-google-auth. 2024-04-06T07:51:56.328 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-06T07:51:56.340 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-04-06T07:51:56.370 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-06T07:51:56.615 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-06T07:51:56.642 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T07:51:56.657 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-06T07:51:56.681 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T07:51:56.684 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-06T07:51:56.715 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T07:51:57.018 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-websocket. 2024-04-06T07:51:57.046 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-04-06T07:51:57.085 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-06T07:51:57.413 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-06T07:51:57.440 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-06T07:51:57.479 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T07:51:57.695 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-babel. 2024-04-06T07:51:57.722 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T07:51:57.761 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T07:51:58.064 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jinja2. 2024-04-06T07:51:58.080 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-06T07:51:58.114 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T07:51:58.185 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-06T07:51:58.213 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:58.252 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:58.426 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-06T07:51:58.442 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:51:58.475 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:51:58.571 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-06T07:51:58.599 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-06T07:51:58.638 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T07:51:58.787 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-06T07:51:58.803 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-06T07:51:58.860 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-06T07:51:58.982 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-06T07:51:59.010 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T07:51:59.048 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T07:51:59.276 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package jq. 2024-04-06T07:51:59.292 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T07:51:59.335 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-06T07:51:59.432 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pluggy. 2024-04-06T07:51:59.459 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-06T07:51:59.498 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-06T07:51:59.587 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package socat. 2024-04-06T07:51:59.604 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-06T07:51:59.645 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-06T07:51:59.793 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-py. 2024-04-06T07:51:59.820 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-04-06T07:51:59.859 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-06T07:51:59.949 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package xmlstarlet. 2024-04-06T07:51:59.965 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-06T07:52:00.007 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-06T07:52:00.213 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-06T07:52:00.240 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-06T07:52:00.279 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-06T07:52:00.387 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-test. 2024-04-06T07:52:00.414 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:00.453 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:00.548 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-toml. 2024-04-06T07:52:00.565 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-04-06T07:52:00.607 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-06T07:52:00.901 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pytest. 2024-04-06T07:52:00.929 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-06T07:52:00.968 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T07:52:01.305 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-06T07:52:01.332 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-04-06T07:52:01.371 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-06T07:52:01.649 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-routes. 2024-04-06T07:52:01.677 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-06T07:52:01.716 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T07:52:02.027 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-06T07:52:02.055 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:52:02.094 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:02.282 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package ceph-volume. 2024-04-06T07:52:02.310 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:52:02.348 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:02.489 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-06T07:52:02.505 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-06T07:52:02.547 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T07:52:02.644 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-06T07:52:02.672 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:02.711 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:02.984 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-joblib. 2024-04-06T07:52:03.005 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package nvme-cli. 2024-04-06T07:52:03.008 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-06T07:52:03.033 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-06T07:52:03.050 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T07:52:03.072 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T07:52:03.396 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-06T07:52:03.418 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-06T07:52:03.423 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-06T07:52:03.434 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-06T07:52:03.462 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-06T07:52:03.468 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-06T07:52:03.732 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn. 2024-04-06T07:52:03.759 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-06T07:52:03.798 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T07:52:03.897 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pastescript. 2024-04-06T07:52:03.924 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-04-06T07:52:03.963 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-06T07:52:04.266 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-psutil. 2024-04-06T07:52:04.282 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-06T07:52:04.316 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-06T07:52:04.523 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-06T07:52:04.551 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:52:04.598 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:04.686 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pygments. 2024-04-06T07:52:04.713 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-06T07:52:04.752 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T07:52:05.181 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-06T07:52:05.209 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-06T07:52:05.264 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-06T07:52:05.542 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package python3-simplejson. 2024-04-06T07:52:05.570 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-06T07:52:05.608 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-06T07:52:05.676 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cachetools. 2024-04-06T07:52:05.703 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-04-06T07:52:05.742 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-06T07:52:05.895 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-06T07:52:05.922 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-06T07:52:05.961 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-06T07:52:06.113 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-06T07:52:06.141 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-06T07:52:06.179 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-06T07:52:06.517 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-06T07:52:06.544 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-06T07:52:06.583 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T07:52:06.648 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package radosgw. 2024-04-06T07:52:06.676 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../107-radosgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:06.706 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:06.894 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rsa. 2024-04-06T07:52:06.922 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-04-06T07:52:06.961 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-06T07:52:07.281 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-google-auth. 2024-04-06T07:52:07.308 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-04-06T07:52:07.347 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-06T07:52:07.451 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package rbd-fuse. 2024-04-06T07:52:07.479 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:07.518 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:07.684 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-06T07:52:07.711 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-06T07:52:07.750 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T07:52:07.862 INFO:teuthology.orchestra.run.smithi129.stdout:Selecting previously unselected package rbd-mirror. 2024-04-06T07:52:07.889 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing to unpack .../109-rbd-mirror_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:07.928 INFO:teuthology.orchestra.run.smithi129.stdout:Unpacking rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:08.003 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-websocket. 2024-04-06T07:52:08.019 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-04-06T07:52:08.053 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-06T07:52:08.365 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-06T07:52:08.392 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-06T07:52:08.431 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T07:52:08.432 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-06T07:52:08.666 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T07:52:08.793 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T07:52:09.055 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-06T07:52:09.103 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-06T07:52:09.132 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:52:09.169 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:09.379 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-06T07:52:09.481 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-06T07:52:09.508 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-06T07:52:09.547 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T07:52:09.825 INFO:teuthology.orchestra.run.smithi129.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-06T07:52:09.892 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-06T07:52:09.920 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T07:52:09.958 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T07:52:10.053 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:10.211 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package jq. 2024-04-06T07:52:10.227 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T07:52:10.269 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-06T07:52:10.522 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package socat. 2024-04-06T07:52:10.536 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user cephadm....done 2024-04-06T07:52:10.549 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-06T07:52:10.588 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-06T07:52:10.875 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package xmlstarlet. 2024-04-06T07:52:10.891 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-06T07:52:10.933 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-06T07:52:10.984 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T07:52:11.260 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-06T07:52:11.278 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-test. 2024-04-06T07:52:11.295 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:11.336 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:11.511 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-06T07:52:11.633 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-06T07:52:11.880 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-06T07:52:12.115 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T07:52:12.241 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-06T07:52:12.543 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T07:52:12.903 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-06T07:52:13.154 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-06T07:52:13.366 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-volume. 2024-04-06T07:52:13.394 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1059-g4d05f4ca-1jammy_all.deb ... 2024-04-06T07:52:13.432 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:13.438 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-06T07:52:13.673 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-06T07:52:13.761 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-06T07:52:13.789 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:13.828 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:13.924 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-06T07:52:14.174 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:14.197 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package nvme-cli. 2024-04-06T07:52:14.225 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-06T07:52:14.264 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T07:52:14.434 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T07:52:14.552 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-06T07:52:14.610 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-06T07:52:14.638 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-06T07:52:14.668 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-06T07:52:14.811 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T07:52:14.937 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-06T07:52:15.071 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastescript. 2024-04-06T07:52:15.099 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-04-06T07:52:15.138 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-06T07:52:15.305 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T07:52:15.431 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-06T07:52:15.433 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-psutil. 2024-04-06T07:52:15.450 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-06T07:52:15.482 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-06T07:52:15.690 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T07:52:15.810 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pygments. 2024-04-06T07:52:15.827 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-06T07:52:15.860 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T07:52:15.986 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-06T07:52:16.272 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-06T07:52:16.301 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-06T07:52:16.318 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T07:52:16.339 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-06T07:52:16.625 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplejson. 2024-04-06T07:52:16.653 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-06T07:52:16.691 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-06T07:52:16.960 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-06T07:52:16.977 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-06T07:52:17.011 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T07:52:17.018 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-06T07:52:17.279 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T07:52:17.389 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-06T07:52:17.531 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T07:52:17.748 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package radosgw. 2024-04-06T07:52:17.776 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../107-radosgw_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:17.814 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:17.833 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-06T07:52:18.217 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-06T07:52:18.468 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T07:52:18.509 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package rbd-fuse. 2024-04-06T07:52:18.525 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:18.567 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:18.845 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T07:52:18.878 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package rbd-mirror. 2024-04-06T07:52:18.906 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../109-rbd-mirror_18.2.2-1059-g4d05f4ca-1jammy_amd64.deb ... 2024-04-06T07:52:18.944 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:19.112 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T07:52:19.447 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-06T07:52:19.524 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-06T07:52:19.698 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-06T07:52:19.758 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T07:52:19.824 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:19.884 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T07:52:20.117 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-idna (3.3-1) ... 2024-04-06T07:52:20.157 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-06T07:52:20.435 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T07:52:20.521 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-06T07:52:20.946 INFO:teuthology.orchestra.run.smithi047.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-06T07:52:21.018 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:21.543 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user cephadm....done 2024-04-06T07:52:21.615 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:52:21.684 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T07:52:21.749 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-06T07:52:22.011 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T07:52:22.036 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-06T07:52:22.137 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-06T07:52:22.263 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T07:52:22.295 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-06T07:52:22.430 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-06T07:52:22.581 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-06T07:52:22.681 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-06T07:52:22.832 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-06T07:52:22.923 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T07:52:23.049 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-06T07:52:23.083 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:52:23.209 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T07:52:23.335 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-06T07:52:23.368 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T07:52:23.628 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-06T07:52:23.728 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-06T07:52:23.871 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-06T07:52:23.970 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-06T07:52:24.130 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-06T07:52:24.247 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-06T07:52:24.364 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-06T07:52:24.506 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-06T07:52:24.616 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-06T07:52:24.765 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-06T07:52:24.891 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T07:52:25.018 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-06T07:52:25.041 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-argparse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:25.294 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T07:52:25.334 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T07:52:25.420 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T07:52:25.460 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-06T07:52:25.704 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-06T07:52:25.722 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T07:52:25.845 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-06T07:52:26.005 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-06T07:52:26.213 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T07:52:26.339 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-06T07:52:26.398 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T07:52:26.607 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T07:52:26.790 INFO:teuthology.orchestra.run.smithi129.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-06T07:52:26.883 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-06T07:52:27.025 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T07:52:27.151 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T07:52:27.226 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T07:52:27.536 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-06T07:52:27.804 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-06T07:52:27.936 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T07:52:28.055 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:52:28.165 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T07:52:28.195 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T07:52:28.321 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-06T07:52:28.447 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T07:52:28.457 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T07:52:28.740 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-06T07:52:28.750 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T07:52:29.018 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-06T07:52:29.124 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-06T07:52:29.143 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-06T07:52:29.376 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T07:52:29.411 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-06T07:52:29.679 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T07:52:29.894 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T07:52:29.930 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-06T07:52:30.187 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T07:52:30.391 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-06T07:52:30.538 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-06T07:52:30.789 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-06T07:52:30.825 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T07:52:30.890 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:31.160 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T07:52:31.182 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-idna (3.3-1) ... 2024-04-06T07:52:31.294 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:31.420 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:31.509 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T07:52:31.546 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:31.664 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:31.765 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:31.985 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-06T07:52:32.261 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:32.387 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:32.513 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:32.714 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T07:52:32.714 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T07:52:32.723 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:52:32.832 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-06T07:52:33.091 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T07:52:33.189 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:33.226 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-06T07:52:33.290 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:33.327 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T07:52:33.416 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T07:52:33.611 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-06T07:52:33.879 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-06T07:52:34.130 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:52:34.248 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-04-06T07:52:34.349 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-06T07:52:34.666 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-06T07:52:34.687 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:34.813 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:34.926 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-06T07:52:34.939 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:35.065 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:35.191 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:35.194 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-06T07:52:35.437 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-06T07:52:35.704 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-06T07:52:35.709 INFO:teuthology.orchestra.run.smithi129.stdout:Adding group ceph....done 2024-04-06T07:52:35.964 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T07:52:36.073 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-06T07:52:36.094 INFO:teuthology.orchestra.run.smithi129.stdout:Adding system user ceph....done 2024-04-06T07:52:36.263 INFO:teuthology.orchestra.run.smithi129.stdout:Setting system user ceph properties....done 2024-04-06T07:52:36.280 INFO:teuthology.orchestra.run.smithi129.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-06T07:52:36.340 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T07:52:36.417 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-06T07:52:36.466 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T07:52:36.752 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-06T07:52:36.768 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-06T07:52:37.061 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-06T07:52:37.251 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:37.303 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T07:52:37.368 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:37.828 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-06T07:52:37.845 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T07:52:37.846 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T07:52:38.081 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T07:52:38.224 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T07:52:38.312 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:38.608 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-06T07:52:38.654 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-06T07:52:38.868 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-06T07:52:39.113 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:39.119 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T07:52:39.245 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T07:52:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T07:52:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T07:52:39.571 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T07:52:39.815 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:39.897 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T07:52:40.018 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T07:52:40.019 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T07:52:40.181 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-06T07:52:40.299 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-06T07:52:40.483 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:40.575 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-06T07:52:40.700 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T07:52:40.701 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T07:52:40.843 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T07:52:41.086 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-06T07:52:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:41.464 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T07:52:41.464 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T07:52:41.478 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-06T07:52:41.771 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T07:52:42.002 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.103 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.106 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T07:52:42.233 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librados2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.254 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.359 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librgw2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.460 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.473 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T07:52:42.473 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T07:52:42.577 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs2 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.703 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libradosstriper1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:42.898 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-06T07:52:42.997 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.166 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librbd1 (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.233 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.276 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-modules-core (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.385 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.391 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.542 INFO:teuthology.orchestra.run.smithi129.stdout:Setting up ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:43.606 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T07:52:43.606 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T07:52:43.912 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-06T07:52:44.103 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs-dev (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:44.254 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rados (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:44.380 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T07:52:44.929 INFO:teuthology.orchestra.run.smithi129.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-06T07:52:45.408 INFO:teuthology.orchestra.run.smithi129.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-06T07:52:45.415 DEBUG:teuthology.parallel:result is None 2024-04-06T07:52:45.676 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rbd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:45.793 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up rbd-fuse (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:45.919 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:46.045 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cephfs (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:46.172 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-common (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:46.723 INFO:teuthology.orchestra.run.smithi047.stdout:Adding group ceph....done 2024-04-06T07:52:47.065 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user ceph....done 2024-04-06T07:52:47.219 INFO:teuthology.orchestra.run.smithi047.stdout:Setting system user ceph properties....done 2024-04-06T07:52:47.238 INFO:teuthology.orchestra.run.smithi047.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-06T07:52:47.380 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-06T07:52:47.705 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-06T07:52:48.147 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-test (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:48.256 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up radosgw (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:48.739 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T07:52:48.739 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T07:52:49.191 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-base (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:49.555 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-06T07:52:50.010 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mds (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:50.249 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T07:52:50.249 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T07:52:50.728 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up rbd-mirror (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:50.945 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T07:52:50.946 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-04-06T07:52:51.438 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:51.660 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T07:52:51.660 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T07:52:52.139 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-osd (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:52.435 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T07:52:52.435 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T07:52:52.932 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:53.067 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:53.218 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mon (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:53.445 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T07:52:53.445 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T07:52:53.944 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-cephadm (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:54.095 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:54.221 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-dashboard (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:54.355 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-volume (18.2.2-1059-g4d05f4ca-1jammy) ... 2024-04-06T07:52:54.708 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-06T07:52:56.001 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-06T07:52:56.485 INFO:teuthology.orchestra.run.smithi047.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-06T07:52:56.490 DEBUG:teuthology.parallel:result is None 2024-04-06T07:52:56.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T07:52:56.720 DEBUG:teuthology.orchestra.run.smithi047:> dpkg-query -W -f '${Version}' ceph 2024-04-06T07:52:56.739 INFO:teuthology.orchestra.run.smithi047.stdout:18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:52:56.740 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:52:56.740 INFO:teuthology.task.install:The correct ceph version 18.2.2-1059-g4d05f4ca-1jammy is installed. 2024-04-06T07:52:56.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4d05f4cad464de0fcdd11a2856643e533aea34e8 2024-04-06T07:52:56.893 DEBUG:teuthology.orchestra.run.smithi129:> dpkg-query -W -f '${Version}' ceph 2024-04-06T07:52:56.912 INFO:teuthology.orchestra.run.smithi129.stdout:18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:52:56.912 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1059-g4d05f4ca-1jammy 2024-04-06T07:52:56.912 INFO:teuthology.task.install:The correct ceph version 18.2.2-1059-g4d05f4ca-1jammy is installed. 2024-04-06T07:52:56.914 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-06T07:52:56.915 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:56.915 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T07:52:56.930 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:52:56.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T07:52:56.970 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-06T07:52:56.970 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:56.971 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T07:52:56.987 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T07:52:57.042 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:52:57.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T07:52:57.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T07:52:57.112 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-06T07:52:57.112 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:57.112 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T07:52:57.126 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T07:52:57.182 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:52:57.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T07:52:57.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T07:52:57.248 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-06T07:52:57.248 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:57.248 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T07:52:57.262 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T07:52:57.318 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:52:57.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T07:52:57.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T07:52:57.384 INFO:teuthology.run_tasks:Running task ceph... 2024-04-06T07:52:57.448 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-06T07:52:57.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2024-04-06T07:52:57.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-04-06T07:52:57.463 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-06T07:52:57.463 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-06T07:52:57.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-06T07:52:57.516 INFO:tasks.ceph:Creating extra log directories... 2024-04-06T07:52:57.516 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-06T07:52:57.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-06T07:52:57.586 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-06T07:52:57.586 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-06T07:52:57.586 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7637392', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9400, 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8', 'sleep_before_teardown': 0, 'subset': '12/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTHMeTm4aEYxXTztGJp/ZnBWzFTNmzWr6I2B05DKL+lPIWvSFoRnUfJ4TA4EGUpMSupoLkseaDMlft5b+aCxmc=', 'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEK2+aO2cc17S44L4cN0L8KoZ3ZphY9k5HggEzkX1MzM4Z8bL4MNfCMihKsDKb7zUo6rjGLJC0m70T94BwUN50='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-06T07:52:57.587 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-06T07:52:57.616 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-06T07:52:57.621 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T07:52:57.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T07:52:57.677 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:57.677 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T07:52:57.723 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T07:52:57.723 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-04-06T07:52:57.771 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T07:52:57.771 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:52:57.771 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-04-06T07:52:57.771 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:52:57.771 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-06 07:47:43.007419034 +0000 2024-04-06T07:52:57.771 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-06 07:47:42.719421606 +0000 2024-04-06T07:52:57.772 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-06 07:47:42.719421606 +0000 2024-04-06T07:52:57.772 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-04-06T07:52:57.772 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T07:52:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-06T07:52:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-06T07:52:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000206324 s, 2.5 MB/s 2024-04-06T07:52:57.826 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T07:52:57.877 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 896 Links: 1 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-06 07:47:43.527414389 +0000 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-06 07:47:43.231417033 +0000 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-06 07:47:43.231417033 +0000 2024-04-06T07:52:57.923 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-04-06T07:52:57.924 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T07:52:57.982 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-06T07:52:57.982 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-06T07:52:57.982 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000320716 s, 1.6 MB/s 2024-04-06T07:52:57.983 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T07:52:58.033 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-06 07:47:44.019409994 +0000 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-06 07:47:43.755412352 +0000 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-06 07:47:43.755412352 +0000 2024-04-06T07:52:58.079 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-04-06T07:52:58.079 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T07:52:58.134 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-06T07:52:58.134 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-06T07:52:58.134 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000300948 s, 1.7 MB/s 2024-04-06T07:52:58.135 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T07:52:58.185 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-04-06T07:52:58.231 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T07:52:58.231 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:52:58.231 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-04-06T07:52:58.231 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:52:58.231 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-04-06 07:47:44.539405349 +0000 2024-04-06T07:52:58.232 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-04-06 07:47:44.243407994 +0000 2024-04-06T07:52:58.232 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-04-06 07:47:44.247407958 +0000 2024-04-06T07:52:58.232 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-04-06T07:52:58.232 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T07:52:58.285 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-04-06T07:52:58.285 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-04-06T07:52:58.286 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000292824 s, 1.7 MB/s 2024-04-06T07:52:58.287 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T07:52:58.340 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:52:58.340 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T07:52:58.340 INFO:tasks.ceph:Generating config... 2024-04-06T07:52:58.341 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-06T07:52:58.341 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client] log to stderr = False 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.342 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.343 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-06T07:52:58.343 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-06T07:52:58.343 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-06T07:52:58.343 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-06T07:52:58.343 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-06T07:52:58.343 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-06T07:52:58.343 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-06T07:52:58.343 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-06T07:52:58.343 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-06T07:52:58.344 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-06T07:52:58.344 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-06T07:52:58.344 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-06T07:52:58.413 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/cluster1.keyring 2024-04-06T07:52:58.431 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-06T07:52:58.510 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-06T07:52:58.566 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.47')] 2024-04-06T07:52:58.567 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.47', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-06T07:52:58.568 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:58.568 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T07:52:58.614 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.47 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: generated fsid b40bb857-7ded-43c7-8c1c-38011a993fe5 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = pacific 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 0 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:fsid b40bb857-7ded-43c7-8c1c-38011a993fe5 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:last_changed 2024-04-06T07:52:58.692467+0000 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:created 2024-04-06T07:52:58.692467+0000 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:min_mon_release 16 (pacific) 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:election_strategy: 1 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-04-06T07:52:58.695 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-06T07:52:58.697 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T07:52:58.743 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID b40bb857-7ded-43c7-8c1c-38011a993fe5... 2024-04-06T07:52:58.744 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-06T07:52:58.788 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-06T07:52:58.816 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-04-06T07:52:58.816 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-04-06T07:52:58.816 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T07:52:58.816 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-04-06T07:52:58.816 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.817 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T07:52:58.818 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = 172.21.15.47 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon client directed command retry = 5 2024-04-06T07:52:58.819 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on pool no app = False 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject socket failures = 5000 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = b40bb857-7ded-43c7-8c1c-38011a993fe5 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.820 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-04-06T07:52:58.821 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: bluefs allocator = hybrid 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = hybrid 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T07:52:58.822 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.823 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T07:52:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T07:52:58.825 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: debug rbd = 20 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: debug rbd_mirror = 15 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: log to stderr = False 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: rbd default features = 125 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: rbd mirror image policy type = simple 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.0] 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.1] 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.2] 2024-04-06T07:52:58.826 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.827 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.827 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.3] 2024-04-06T07:52:58.827 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.827 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.827 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-04-06T07:52:58.827 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.828 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-04-06T07:52:58.829 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-04-06T07:52:58.830 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = 172.21.15.47 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: mon client directed command retry = 5 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on pool no app = False 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject socket failures = 5000 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = b40bb857-7ded-43c7-8c1c-38011a993fe5 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.831 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-04-06T07:52:58.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: bluefs allocator = hybrid 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = hybrid 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-04-06T07:52:58.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:52:58.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:52:58.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-04-06T07:52:58.836 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd = 20 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd_mirror = 15 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: log to stderr = False 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: rbd default features = 125 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: rbd mirror image policy type = simple 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.0] 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.1] 2024-04-06T07:52:58.837 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.2] 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.3] 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:52:58.838 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:52:58.839 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-06T07:52:58.840 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-06T07:52:58.931 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-06T07:52:59.238 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:59.238 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-06T07:52:59.244 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:59.244 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-06T07:52:59.291 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T07:52:59.292 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:59.292 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-06T07:52:59.292 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-06T07:52:59.356 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:52:59.356 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:52:59.403 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:52:59.403 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:52:59.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-06T07:52:59.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-06T07:52:59.425 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:52:59.425 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:52:59.473 INFO:tasks.ceph:Setting up mon nodes... 2024-04-06T07:52:59.473 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-06T07:52:59.473 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-06T07:52:59.520 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-06T07:52:59.523 INFO:tasks.ceph:Setting up mds nodes... 2024-04-06T07:52:59.523 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-06T07:52:59.523 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-04-06T07:52:59.556 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-04-06T07:52:59.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-06T07:52:59.601 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-06T07:52:59.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-06T07:52:59.700 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-06T07:52:59.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-06T07:52:59.784 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-06T07:52:59.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-06T07:52:59.866 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-06T07:52:59.876 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-06T07:52:59.956 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-06T07:52:59.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-06T07:53:00.038 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-06T07:53:00.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-06T07:53:00.115 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-06T07:53:00.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-06T07:53:00.197 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-06T07:53:00.206 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-06T07:53:00.206 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T07:53:00.206 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-06T07:53:00.220 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:00.220 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-06T07:53:00.220 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T07:53:00.220 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T07:53:00.278 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T07:53:00.304 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-04-06T07:53:00.306 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-04-06T07:53:00.306 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-06T07:53:00.368 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-06T07:53:00.420 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T07:53:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:53:00.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-06T07:53:00.479 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:00.479 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-06T07:53:00.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T07:53:00.479 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T07:53:00.535 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T07:53:00.562 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-04-06T07:53:00.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-04-06T07:53:00.564 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-06T07:53:00.624 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-06T07:53:00.676 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T07:53:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:53:00.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-06T07:53:00.730 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:00.730 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-06T07:53:00.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T07:53:00.731 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-06T07:53:00.787 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T07:53:00.788 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T07:53:00.814 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-04-06T07:53:00.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-04-06T07:53:00.816 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-06T07:53:00.874 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-06T07:53:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:53:00.928 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T07:53:00.928 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:53:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:01.033+0000 7fb1b552b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-06T07:53:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:01.033+0000 7fb1b552b640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-06T07:53:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:01.033+0000 7fb1b552b640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-06T07:53:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:01.033+0000 7fb1b552b640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-06T07:53:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:01.033+0000 7fb1b552b640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-06T07:53:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:01.033+0000 7fb1b552b640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-06T07:53:04.023 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-06T07:53:04.037 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:53:04.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:04.109+0000 7f26650ab640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-06T07:53:04.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:04.109+0000 7f26650ab640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-06T07:53:04.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:04.109+0000 7f26650ab640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-06T07:53:04.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:04.109+0000 7f26650ab640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-06T07:53:04.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:04.109+0000 7f26650ab640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-06T07:53:04.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:04.109+0000 7f26650ab640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-06T07:53:07.077 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-06T07:53:07.131 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:53:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:07.209+0000 7f81b11ea640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-06T07:53:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:07.209+0000 7f81b11ea640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-06T07:53:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:07.209+0000 7f81b11ea640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-06T07:53:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:07.209+0000 7f81b11ea640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-06T07:53:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:07.209+0000 7f81b11ea640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-06T07:53:07.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-04-06T07:53:07.209+0000 7f81b11ea640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-06T07:53:10.207 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-06T07:53:10.263 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-06T07:53:10.263 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:10.263 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-06T07:53:10.318 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:10.318 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-06T07:53:10.374 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:10.374 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-06T07:53:10.430 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:10.430 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-06T07:53:10.487 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:10.487 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-04-06T07:53:10.539 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.539 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-06T07:53:10.545 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.546 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-06T07:53:10.593 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.593 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-04-06T07:53:10.641 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.641 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-04-06T07:53:10.688 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.689 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-04-06T07:53:10.737 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.737 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-04-06T07:53:10.785 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.785 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-04-06T07:53:10.833 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:10.833 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-04-06T07:53:10.880 INFO:tasks.ceph:Adding keys to all mons... 2024-04-06T07:53:10.881 DEBUG:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBmHQ7eHhAAdsuGBH+FzW+huwc4JE1Oyw== 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBd/xBmgtoqAhAAt8V4OjLmSfCzBUC/aEgyYA== 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBg/xBmBNvWBhAAizBhr8MJ8KMMTX1bX4Xg+g== 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBj/xBmuGOqDBAA25qS9N9h74MbcvJv6eGGBg== 2024-04-06T07:53:10.893 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBmuoAJIRAAoHWjYe6QecVbYcZ/dg88iw== 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBm41K8IxAAW+aj1np3leFLsPAhCTqmiA== 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.0] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBm64OqKRAAHjjyaT90RL8Nm7xoWR6IDA== 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.1] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBm/pqiLhAAMVwbX8Mv3R9kYO4pVHIjVw== 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.2] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBmFemTMxAAX6Xhh5LLlr81eDOI3wgEow== 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.3] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBb/xBmthftOBAAazdwB8y8xWIOHxOYl0yNUg== 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.4] 2024-04-06T07:53:10.894 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBc/xBmuzYuAhAA1mIyd/Iom/8Six/hQCcXHw== 2024-04-06T07:53:10.895 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.5] 2024-04-06T07:53:10.895 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBc/xBm1QLCBhAArnMNL5QzuQXp6Ll4lkgbtQ== 2024-04-06T07:53:10.895 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.6] 2024-04-06T07:53:10.895 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQBc/xBmJv6mCxAAKhIQCVx2VLKx9NzDCWfyPQ== 2024-04-06T07:53:10.896 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T07:53:10.984 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T07:53:11.026 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T07:53:11.110 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T07:53:11.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.289 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.500 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.590 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.636 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.759 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:11.844 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-06T07:53:11.844 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-06T07:53:11.896 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-06T07:53:11.984 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-06T07:53:12.036 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-06T07:53:12.082 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-06T07:53:12.082 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-06T07:53:12.082 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-06T07:53:12.127 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-06T07:53:12.127 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-06T07:53:12.128 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-06T07:53:12.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-06T07:53:12.129 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-06T07:53:12.130 INFO:tasks.ceph:Setting crush tunables to default 2024-04-06T07:53:12.130 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-06T07:53:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-04-06T07:53:12.385 INFO:tasks.ceph:check_enable_crimson: False 2024-04-06T07:53:12.385 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-06T07:53:12.385 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:12.385 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-06T07:53:12.395 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:12.395 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-06T07:53:12.452 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:12.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-06T07:53:12.507 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 osd new 4c374052-d1fe-488f-878f-354cadfee4ac 0 2024-04-06T07:53:12.510 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:12.505+0000 7fc830eb3180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T07:53:12.712 INFO:teuthology.orchestra.run.smithi047.stdout:0 2024-04-06T07:53:12.730 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 osd new 84cff3d1-8b34-4a58-bffc-a4cc844f7212 1 2024-04-06T07:53:12.932 INFO:teuthology.orchestra.run.smithi047.stdout:1 2024-04-06T07:53:12.949 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 osd new 7369452f-d483-4735-928d-aa461b4cffb1 2 2024-04-06T07:53:13.017 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:13.009+0000 7fc830eb3180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T07:53:13.141 INFO:teuthology.orchestra.run.smithi047.stdout:2 2024-04-06T07:53:13.157 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-06T07:53:13.157 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-06T07:53:13.159 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-06T07:53:13.160 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-06T07:53:13.160 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-06T07:53:13.162 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-06T07:53:13.162 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-06T07:53:13.162 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-06T07:53:13.164 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-06T07:53:13.164 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T07:53:13.228 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:13.221+0000 7fc830eb3180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T07:53:13.351 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:13.351 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":5,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T07:53:13.136443+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:13.366 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-06T07:53:13.367 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-06T07:53:13.669 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-06T07:53:13.767 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T07:53:13.761+0000 7f04f17c5640 -1 Falling back to public interface 2024-04-06T07:53:13.791 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T07:53:13.785+0000 7f48f55ff640 -1 Falling back to public interface 2024-04-06T07:53:13.799 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T07:53:13.793+0000 7f58ad6a4640 -1 Falling back to public interface 2024-04-06T07:53:13.809 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:13.801+0000 7fc830eb3180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T07:53:13.855 INFO:teuthology.misc.health.smithi047.stdout: 2024-04-06T07:53:13.855 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":5,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T07:53:13.136443+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:13.865 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:13.857+0000 7fc830eb3180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T07:53:13.870 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T07:53:13.920 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:13.913+0000 7fc830eb3180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T07:53:14.000 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:13.993+0000 7fc830eb3180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T07:53:14.053 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:14.045+0000 7fc830eb3180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T07:53:14.133 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:14.125+0000 7fc830eb3180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T07:53:14.186 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:14.181+0000 7fc830eb3180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T07:53:14.342 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:14.333+0000 7fc830eb3180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T07:53:14.405 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:14.397+0000 7fc830eb3180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T07:53:15.145 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-06T07:53:15.146 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-06T07:53:15.146 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr: from numpy import show_config as show_numpy_config 2024-04-06T07:53:15.208 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:15.201+0000 7fc830eb3180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T07:53:15.378 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:15.369+0000 7fc830eb3180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T07:53:15.544 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:15.537+0000 7fc830eb3180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T07:53:15.615 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:15.609+0000 7fc830eb3180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T07:53:15.764 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:15.757+0000 7fc830eb3180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T07:53:16.408 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:16.401+0000 7fc830eb3180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T07:53:16.486 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:16.477+0000 7fc830eb3180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T07:53:16.679 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:16.673+0000 7fc830eb3180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T07:53:16.755 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:16.749+0000 7fc830eb3180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T07:53:17.116 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:17.109+0000 7fc830eb3180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T07:53:17.180 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:17.173+0000 7fc830eb3180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T07:53:17.634 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T07:53:17.629+0000 7f04f17c5640 -1 osd.1 0 log_to_monitors true 2024-04-06T07:53:17.642 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T07:53:17.637+0000 7f48f55ff640 -1 osd.2 0 log_to_monitors true 2024-04-06T07:53:17.655 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T07:53:17.649+0000 7f58ad6a4640 -1 osd.0 0 log_to_monitors true 2024-04-06T07:53:18.424 INFO:tasks.ceph.cluster1.mgr.x.smithi047.stderr:2024-04-06T07:53:18.417+0000 7fc7d856a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-06T07:53:20.173 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-06T07:53:20.677 INFO:teuthology.misc.health.smithi047.stdout: 2024-04-06T07:53:20.677 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":7,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T07:53:19.255385+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:20.701 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T07:53:20.701 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T07:53:20.693+0000 7f04edf81640 -1 osd.1 0 waiting for initial osdmap 2024-04-06T07:53:20.743 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T07:53:20.737+0000 7f04e559d640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T07:53:20.772 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T07:53:20.765+0000 7f58a9649640 -1 osd.0 0 waiting for initial osdmap 2024-04-06T07:53:20.772 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T07:53:20.765+0000 7f48f15a4640 -1 osd.2 0 waiting for initial osdmap 2024-04-06T07:53:20.807 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T07:53:20.801+0000 7f58a147c640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T07:53:20.807 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T07:53:20.801+0000 7f48e93d7640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T07:53:27.004 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-06T07:53:27.306 INFO:teuthology.misc.health.smithi047.stdout: 2024-04-06T07:53:27.306 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":12,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T07:53:25.718104+0000","last_up_change":"2024-04-06T07:53:21.704210+0000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:23.281297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":23782},{"type":"v1","addr":"172.21.15.47:6817","nonce":23782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":23782},{"type":"v1","addr":"172.21.15.47:6819","nonce":23782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":23782},{"type":"v1","addr":"172.21.15.47:6823","nonce":23782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":23782},{"type":"v1","addr":"172.21.15.47:6821","nonce":23782}]},"public_addr":"172.21.15.47:6817/23782","cluster_addr":"172.21.15.47:6819/23782","heartbeat_back_addr":"172.21.15.47:6823/23782","heartbeat_front_addr":"172.21.15.47:6821/23782","state":["exists","up"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":23778},{"type":"v1","addr":"172.21.15.47:6801","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":23778},{"type":"v1","addr":"172.21.15.47:6803","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":23778},{"type":"v1","addr":"172.21.15.47:6807","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":23778},{"type":"v1","addr":"172.21.15.47:6805","nonce":23778}]},"public_addr":"172.21.15.47:6801/23778","cluster_addr":"172.21.15.47:6803/23778","heartbeat_back_addr":"172.21.15.47:6807/23778","heartbeat_front_addr":"172.21.15.47:6805/23778","state":["exists","up"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":23779},{"type":"v1","addr":"172.21.15.47:6809","nonce":23779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":23779},{"type":"v1","addr":"172.21.15.47:6811","nonce":23779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":23779},{"type":"v1","addr":"172.21.15.47:6815","nonce":23779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":23779},{"type":"v1","addr":"172.21.15.47:6813","nonce":23779}]},"public_addr":"172.21.15.47:6809/23779","cluster_addr":"172.21.15.47:6811/23779","heartbeat_back_addr":"172.21.15.47:6815/23779","heartbeat_front_addr":"172.21.15.47:6813/23779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.630582+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.635946+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.596933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:27.324 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-06T07:53:27.324 INFO:tasks.ceph:Creating RBD pool 2024-04-06T07:53:27.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-06T07:53:28.220 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'rbd' created 2024-04-06T07:53:28.245 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-06T07:53:29.225 INFO:teuthology.orchestra.run.smithi047.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-06T07:53:29.255 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-06T07:53:29.256 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-06T07:53:29.256 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-06T07:53:29.256 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-06T07:53:29.256 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-06T07:53:29.584 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:29.601 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":23602},{"type":"v1","addr":"172.21.15.47:6825","nonce":23602}]},"active_addr":"172.21.15.47:6825/23602","active_change":"2024-04-06T07:53:17.197136+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":20958910}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":682461964}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":113650631}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":2854417783}]}]} 2024-04-06T07:53:29.603 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-06T07:53:29.603 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-06T07:53:29.603 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T07:53:29.892 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:29.892 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":14,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T07:53:29.213784+0000","last_up_change":"2024-04-06T07:53:21.704210+0000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:23.281297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T07:53:27.663437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":23782},{"type":"v1","addr":"172.21.15.47:6817","nonce":23782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":23782},{"type":"v1","addr":"172.21.15.47:6819","nonce":23782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":23782},{"type":"v1","addr":"172.21.15.47:6823","nonce":23782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":23782},{"type":"v1","addr":"172.21.15.47:6821","nonce":23782}]},"public_addr":"172.21.15.47:6817/23782","cluster_addr":"172.21.15.47:6819/23782","heartbeat_back_addr":"172.21.15.47:6823/23782","heartbeat_front_addr":"172.21.15.47:6821/23782","state":["exists","up"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":23778},{"type":"v1","addr":"172.21.15.47:6801","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":23778},{"type":"v1","addr":"172.21.15.47:6803","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":23778},{"type":"v1","addr":"172.21.15.47:6807","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":23778},{"type":"v1","addr":"172.21.15.47:6805","nonce":23778}]},"public_addr":"172.21.15.47:6801/23778","cluster_addr":"172.21.15.47:6803/23778","heartbeat_back_addr":"172.21.15.47:6807/23778","heartbeat_front_addr":"172.21.15.47:6805/23778","state":["exists","up"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":23779},{"type":"v1","addr":"172.21.15.47:6809","nonce":23779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":23779},{"type":"v1","addr":"172.21.15.47:6811","nonce":23779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":23779},{"type":"v1","addr":"172.21.15.47:6815","nonce":23779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":23779},{"type":"v1","addr":"172.21.15.47:6813","nonce":23779}]},"public_addr":"172.21.15.47:6809/23779","cluster_addr":"172.21.15.47:6811/23779","heartbeat_back_addr":"172.21.15.47:6815/23779","heartbeat_front_addr":"172.21.15.47:6813/23779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.630582+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.635946+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.596933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:29.909 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-06T07:53:29.909 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T07:53:30.194 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:30.194 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":14,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T07:53:29.213784+0000","last_up_change":"2024-04-06T07:53:21.704210+0000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:23.281297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T07:53:27.663437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":23782},{"type":"v1","addr":"172.21.15.47:6817","nonce":23782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":23782},{"type":"v1","addr":"172.21.15.47:6819","nonce":23782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":23782},{"type":"v1","addr":"172.21.15.47:6823","nonce":23782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":23782},{"type":"v1","addr":"172.21.15.47:6821","nonce":23782}]},"public_addr":"172.21.15.47:6817/23782","cluster_addr":"172.21.15.47:6819/23782","heartbeat_back_addr":"172.21.15.47:6823/23782","heartbeat_front_addr":"172.21.15.47:6821/23782","state":["exists","up"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":23778},{"type":"v1","addr":"172.21.15.47:6801","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":23778},{"type":"v1","addr":"172.21.15.47:6803","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":23778},{"type":"v1","addr":"172.21.15.47:6807","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":23778},{"type":"v1","addr":"172.21.15.47:6805","nonce":23778}]},"public_addr":"172.21.15.47:6801/23778","cluster_addr":"172.21.15.47:6803/23778","heartbeat_back_addr":"172.21.15.47:6807/23778","heartbeat_front_addr":"172.21.15.47:6805/23778","state":["exists","up"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":23779},{"type":"v1","addr":"172.21.15.47:6809","nonce":23779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":23779},{"type":"v1","addr":"172.21.15.47:6811","nonce":23779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":23779},{"type":"v1","addr":"172.21.15.47:6815","nonce":23779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":23779},{"type":"v1","addr":"172.21.15.47:6813","nonce":23779}]},"public_addr":"172.21.15.47:6809/23779","cluster_addr":"172.21.15.47:6811/23779","heartbeat_back_addr":"172.21.15.47:6815/23779","heartbeat_front_addr":"172.21.15.47:6813/23779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.630582+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.635946+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.596933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:30.213 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-06T07:53:30.213 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-06T07:53:30.214 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-06T07:53:30.363 INFO:teuthology.orchestra.run.smithi047.stdout:34359738372 2024-04-06T07:53:30.364 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-06T07:53:30.382 INFO:teuthology.orchestra.run.smithi047.stdout:34359738373 2024-04-06T07:53:30.382 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-06T07:53:30.385 INFO:teuthology.orchestra.run.smithi047.stdout:34359738373 2024-04-06T07:53:30.385 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-06T07:53:30.660 INFO:teuthology.orchestra.run.smithi047.stdout:34359738371 2024-04-06T07:53:30.677 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2024-04-06T07:53:30.713 INFO:teuthology.orchestra.run.smithi047.stdout:34359738372 2024-04-06T07:53:30.720 INFO:teuthology.orchestra.run.smithi047.stdout:34359738372 2024-04-06T07:53:30.732 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2024-04-06T07:53:30.738 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-04-06T07:53:31.678 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-06T07:53:31.734 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-06T07:53:31.738 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-06T07:53:31.992 INFO:teuthology.orchestra.run.smithi047.stdout:34359738372 2024-04-06T07:53:32.009 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2024-04-06T07:53:32.009 DEBUG:teuthology.parallel:result is None 2024-04-06T07:53:32.026 INFO:teuthology.orchestra.run.smithi047.stdout:34359738373 2024-04-06T07:53:32.043 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-04-06T07:53:32.043 DEBUG:teuthology.parallel:result is None 2024-04-06T07:53:32.077 INFO:teuthology.orchestra.run.smithi047.stdout:34359738373 2024-04-06T07:53:32.094 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-04-06T07:53:32.094 DEBUG:teuthology.parallel:result is None 2024-04-06T07:53:32.094 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-06T07:53:32.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T07:53:32.420 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:32.421 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T07:53:32.438 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T07:53:31.209508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94868,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":93307,"kb_avail":283020652,"statfs":{"total":289910292480,"available":289813147648,"internally_reserved":0,"allocated":1556480,"data_stored":1281896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":95546706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.995430"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236858+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T07:53:29.236858+0000","last_peered":"2024-04-06T07:53:29.236858+0000","last_clean":"2024-04-06T07:53:29.236858+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T07:53:29.236858+0000","last_undegraded":"2024-04-06T07:53:29.236858+0000","last_fullsized":"2024-04-06T07:53:29.236858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:44.700586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236257+0000","last_change":"2024-04-06T07:53:29.233161+0000","last_active":"2024-04-06T07:53:29.236257+0000","last_peered":"2024-04-06T07:53:29.236257+0000","last_clean":"2024-04-06T07:53:29.236257+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T07:53:29.236257+0000","last_undegraded":"2024-04-06T07:53:29.236257+0000","last_fullsized":"2024-04-06T07:53:29.236257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:34.387543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236876+0000","last_change":"2024-04-06T07:53:29.235401+0000","last_active":"2024-04-06T07:53:29.236876+0000","last_peered":"2024-04-06T07:53:29.236876+0000","last_clean":"2024-04-06T07:53:29.236876+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T07:53:29.236876+0000","last_undegraded":"2024-04-06T07:53:29.236876+0000","last_fullsized":"2024-04-06T07:53:29.236876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:05.919002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236307+0000","last_change":"2024-04-06T07:53:29.233180+0000","last_active":"2024-04-06T07:53:29.236307+0000","last_peered":"2024-04-06T07:53:29.236307+0000","last_clean":"2024-04-06T07:53:29.236307+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T07:53:29.236307+0000","last_undegraded":"2024-04-06T07:53:29.236307+0000","last_fullsized":"2024-04-06T07:53:29.236307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:51:57.674391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.234608+0000","last_change":"2024-04-06T07:53:29.231074+0000","last_active":"2024-04-06T07:53:29.234608+0000","last_peered":"2024-04-06T07:53:29.234608+0000","last_clean":"2024-04-06T07:53:29.234608+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T07:53:29.234608+0000","last_undegraded":"2024-04-06T07:53:29.234608+0000","last_fullsized":"2024-04-06T07:53:29.234608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:19:19.458982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T07:53:30.225041+0000","last_change":"2024-04-06T07:53:29.233823+0000","last_active":"2024-04-06T07:53:30.225041+0000","last_peered":"2024-04-06T07:53:30.225041+0000","last_clean":"2024-04-06T07:53:30.225041+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T07:53:30.225041+0000","last_undegraded":"2024-04-06T07:53:30.225041+0000","last_fullsized":"2024-04-06T07:53:30.225041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:37.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T07:53:30.225025+0000","last_change":"2024-04-06T07:53:29.231274+0000","last_active":"2024-04-06T07:53:30.225025+0000","last_peered":"2024-04-06T07:53:30.225025+0000","last_clean":"2024-04-06T07:53:30.225025+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T07:53:30.225025+0000","last_undegraded":"2024-04-06T07:53:30.225025+0000","last_fullsized":"2024-04-06T07:53:30.225025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:24.362133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.234071+0000","last_change":"2024-04-06T07:53:29.232232+0000","last_active":"2024-04-06T07:53:29.234071+0000","last_peered":"2024-04-06T07:53:29.234071+0000","last_clean":"2024-04-06T07:53:29.234071+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T07:53:29.234071+0000","last_undegraded":"2024-04-06T07:53:29.234071+0000","last_fullsized":"2024-04-06T07:53:29.234071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:23.930116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:53:28.224697+0000","last_change":"2024-04-06T07:53:24.763344+0000","last_active":"2024-04-06T07:53:28.224697+0000","last_peered":"2024-04-06T07:53:28.224697+0000","last_clean":"2024-04-06T07:53:28.224697+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T07:53:28.224697+0000","last_undegraded":"2024-04-06T07:53:28.224697+0000","last_fullsized":"2024-04-06T07:53:28.224697+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_clean_scrub_stamp":"2024-04-06T07:53:23.707131+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:54.646711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31236,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340604,"statfs":{"total":96636764160,"available":96604778496,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T07:53:32.439 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T07:53:32.718 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:32.718 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T07:53:32.735 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T07:53:31.209508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":94868,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":93307,"kb_avail":283020652,"statfs":{"total":289910292480,"available":289813147648,"internally_reserved":0,"allocated":1556480,"data_stored":1281896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":95546706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.995430"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236858+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T07:53:29.236858+0000","last_peered":"2024-04-06T07:53:29.236858+0000","last_clean":"2024-04-06T07:53:29.236858+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T07:53:29.236858+0000","last_undegraded":"2024-04-06T07:53:29.236858+0000","last_fullsized":"2024-04-06T07:53:29.236858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:44.700586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236257+0000","last_change":"2024-04-06T07:53:29.233161+0000","last_active":"2024-04-06T07:53:29.236257+0000","last_peered":"2024-04-06T07:53:29.236257+0000","last_clean":"2024-04-06T07:53:29.236257+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T07:53:29.236257+0000","last_undegraded":"2024-04-06T07:53:29.236257+0000","last_fullsized":"2024-04-06T07:53:29.236257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:34.387543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236876+0000","last_change":"2024-04-06T07:53:29.235401+0000","last_active":"2024-04-06T07:53:29.236876+0000","last_peered":"2024-04-06T07:53:29.236876+0000","last_clean":"2024-04-06T07:53:29.236876+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T07:53:29.236876+0000","last_undegraded":"2024-04-06T07:53:29.236876+0000","last_fullsized":"2024-04-06T07:53:29.236876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:05.919002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.236307+0000","last_change":"2024-04-06T07:53:29.233180+0000","last_active":"2024-04-06T07:53:29.236307+0000","last_peered":"2024-04-06T07:53:29.236307+0000","last_clean":"2024-04-06T07:53:29.236307+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T07:53:29.236307+0000","last_undegraded":"2024-04-06T07:53:29.236307+0000","last_fullsized":"2024-04-06T07:53:29.236307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:51:57.674391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.234608+0000","last_change":"2024-04-06T07:53:29.231074+0000","last_active":"2024-04-06T07:53:29.234608+0000","last_peered":"2024-04-06T07:53:29.234608+0000","last_clean":"2024-04-06T07:53:29.234608+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T07:53:29.234608+0000","last_undegraded":"2024-04-06T07:53:29.234608+0000","last_fullsized":"2024-04-06T07:53:29.234608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:19:19.458982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T07:53:30.225041+0000","last_change":"2024-04-06T07:53:29.233823+0000","last_active":"2024-04-06T07:53:30.225041+0000","last_peered":"2024-04-06T07:53:30.225041+0000","last_clean":"2024-04-06T07:53:30.225041+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T07:53:30.225041+0000","last_undegraded":"2024-04-06T07:53:30.225041+0000","last_fullsized":"2024-04-06T07:53:30.225041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:37.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-06T07:53:30.225025+0000","last_change":"2024-04-06T07:53:29.231274+0000","last_active":"2024-04-06T07:53:30.225025+0000","last_peered":"2024-04-06T07:53:30.225025+0000","last_clean":"2024-04-06T07:53:30.225025+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T07:53:30.225025+0000","last_undegraded":"2024-04-06T07:53:30.225025+0000","last_fullsized":"2024-04-06T07:53:30.225025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:24.362133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:53:29.234071+0000","last_change":"2024-04-06T07:53:29.232232+0000","last_active":"2024-04-06T07:53:29.234071+0000","last_peered":"2024-04-06T07:53:29.234071+0000","last_clean":"2024-04-06T07:53:29.234071+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T07:53:29.234071+0000","last_undegraded":"2024-04-06T07:53:29.234071+0000","last_fullsized":"2024-04-06T07:53:29.234071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:23.930116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:53:28.224697+0000","last_change":"2024-04-06T07:53:24.763344+0000","last_active":"2024-04-06T07:53:28.224697+0000","last_peered":"2024-04-06T07:53:28.224697+0000","last_clean":"2024-04-06T07:53:28.224697+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T07:53:28.224697+0000","last_undegraded":"2024-04-06T07:53:28.224697+0000","last_fullsized":"2024-04-06T07:53:28.224697+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_clean_scrub_stamp":"2024-04-06T07:53:23.707131+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:54.646711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31236,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340604,"statfs":{"total":96636764160,"available":96604778496,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31816,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T07:53:32.736 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-06T07:53:32.736 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-06T07:53:32.736 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-06T07:53:32.736 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-06T07:53:33.043 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:33.043 INFO:teuthology.orchestra.run.smithi047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T07:53:33.060 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-06T07:53:33.061 INFO:teuthology.run_tasks:Running task ceph... 2024-04-06T07:53:33.070 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-06T07:53:33.070 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-06T07:53:33.070 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392', 'branch': 'reef', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7637392', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9400, 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8', 'sleep_before_teardown': 0, 'subset': '12/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '052adcadd0534eddb0d73c5afbc083a84cd9dc46', 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTHMeTm4aEYxXTztGJp/ZnBWzFTNmzWr6I2B05DKL+lPIWvSFoRnUfJ4TA4EGUpMSupoLkseaDMlft5b+aCxmc=', 'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEK2+aO2cc17S44L4cN0L8KoZ3ZphY9k5HggEzkX1MzM4Z8bL4MNfCMihKsDKb7zUo6rjGLJC0m70T94BwUN50='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '4d05f4cad464de0fcdd11a2856643e533aea34e8'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-06T07:53:33.071 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-06T07:53:33.076 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-06T07:53:33.082 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T07:53:33.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-04-06T07:53:33.137 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:33.137 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T07:53:33.185 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T07:53:33.185 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-06 07:47:40.749263761 +0000 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-06 07:47:40.481266154 +0000 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-06 07:47:40.481266154 +0000 2024-04-06T07:53:33.233 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-04-06T07:53:33.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T07:53:33.287 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-06T07:53:33.287 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-06T07:53:33.287 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000428964 s, 1.2 MB/s 2024-04-06T07:53:33.288 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T07:53:33.339 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-04-06T07:53:33.384 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-06 07:47:41.217259581 +0000 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-06 07:47:40.957261903 +0000 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-06 07:47:40.957261903 +0000 2024-04-06T07:53:33.385 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-04-06T07:53:33.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T07:53:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-06T07:53:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-06T07:53:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000415927 s, 1.2 MB/s 2024-04-06T07:53:33.440 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T07:53:33.490 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-04-06T07:53:33.536 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T07:53:33.536 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:53:33.536 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 911 Links: 1 2024-04-06T07:53:33.536 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:53:33.536 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-06 07:47:41.697255292 +0000 2024-04-06T07:53:33.537 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-06 07:47:41.441257580 +0000 2024-04-06T07:53:33.537 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-06 07:47:41.441257580 +0000 2024-04-06T07:53:33.537 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-04-06T07:53:33.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T07:53:33.591 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-06T07:53:33.591 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-06T07:53:33.591 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000408324 s, 1.3 MB/s 2024-04-06T07:53:33.592 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T07:53:33.642 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-04-06T07:53:33.688 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T07:53:33.688 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T07:53:33.688 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-04-06T07:53:33.688 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T07:53:33.689 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-06 07:47:42.169251077 +0000 2024-04-06T07:53:33.689 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-06 07:47:41.917253327 +0000 2024-04-06T07:53:33.689 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-06 07:47:41.917253327 +0000 2024-04-06T07:53:33.689 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: - 2024-04-06T07:53:33.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T07:53:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-06T07:53:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-06T07:53:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000373291 s, 1.4 MB/s 2024-04-06T07:53:33.744 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T07:53:33.795 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:33.795 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T07:53:33.796 INFO:tasks.ceph:Generating config... 2024-04-06T07:53:33.798 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-06T07:53:33.798 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-06T07:53:33.798 INFO:tasks.ceph:[client] log to stderr = False 2024-04-06T07:53:33.798 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-06T07:53:33.798 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-04-06T07:53:33.798 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:33.799 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:33.799 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:33.799 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:33.799 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:33.799 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:33.799 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:33.800 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:33.800 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-06T07:53:33.800 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-06T07:53:33.800 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-06T07:53:33.800 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-06T07:53:33.800 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-06T07:53:33.801 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-06T07:53:33.801 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-06T07:53:33.801 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-06T07:53:33.801 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-04-06T07:53:33.801 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-04-06T07:53:33.801 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-06T07:53:33.801 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-06T07:53:33.802 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-06T07:53:33.802 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-06T07:53:33.802 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-06T07:53:33.803 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-06T07:53:33.803 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-06T07:53:33.804 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-06T07:53:33.804 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-06T07:53:33.805 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-06T07:53:33.805 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-06T07:53:33.805 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-06T07:53:33.805 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-06T07:53:33.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-06T07:53:33.867 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.keyring 2024-04-06T07:53:33.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-06T07:53:33.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-06T07:53:34.004 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.129')] 2024-04-06T07:53:34.004 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-06T07:53:34.005 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:34.005 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T07:53:34.052 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:34.131 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:34.131 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid fd3eff46-be4e-471a-92ae-a94b26dd3208 2024-04-06T07:53:34.131 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = pacific 2024-04-06T07:53:34.131 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2024-04-06T07:53:34.131 INFO:teuthology.orchestra.run.smithi129.stdout:fsid fd3eff46-be4e-471a-92ae-a94b26dd3208 2024-04-06T07:53:34.131 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2024-04-06T07:53:34.130024+0000 2024-04-06T07:53:34.132 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-04-06T07:53:34.130024+0000 2024-04-06T07:53:34.132 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 16 (pacific) 2024-04-06T07:53:34.132 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 1 2024-04-06T07:53:34.132 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-04-06T07:53:34.132 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-06T07:53:34.133 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-06T07:53:34.181 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID fd3eff46-be4e-471a-92ae-a94b26dd3208... 2024-04-06T07:53:34.181 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-06T07:53:34.184 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-06T07:53:34.226 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.227 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-04-06T07:53:34.228 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-04-06T07:53:34.229 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = 172.21.15.129 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: mon client directed command retry = 5 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on pool no app = False 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject socket failures = 5000 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = fd3eff46-be4e-471a-92ae-a94b26dd3208 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.230 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-04-06T07:53:34.231 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: bluefs allocator = hybrid 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = hybrid 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1/20 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1/20 2024-04-06T07:53:34.232 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 4/10 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:53:34.233 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.234 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-04-06T07:53:34.235 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: debug rbd = 20 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: debug rbd_mirror = 15 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: log to stderr = False 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: rbd default features = 125 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: rbd mirror image policy type = simple 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.0] 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.1] 2024-04-06T07:53:34.236 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.2] 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout:[client.mirror.3] 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.237 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.254 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-04-06T07:53:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-04-06T07:53:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = 172.21.15.129 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: mon client directed command retry = 5 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on pool no app = False 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject socket failures = 5000 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = fd3eff46-be4e-471a-92ae-a94b26dd3208 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-04-06T07:53:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-04-06T07:53:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: bluefs allocator = hybrid 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = hybrid 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-06T07:53:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:53:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-04-06T07:53:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd = 20 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd_mirror = 15 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: log to stderr = False 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: rbd default features = 125 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: rbd mirror image policy type = simple 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.0] 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.1] 2024-04-06T07:53:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.2] 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.3] 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-06T07:53:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-06T07:53:34.266 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-06T07:53:34.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-06T07:53:34.355 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-06T07:53:34.355 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:34.355 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-06T07:53:34.361 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:34.361 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-06T07:53:34.408 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T07:53:34.408 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:34.408 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-06T07:53:34.408 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-06T07:53:34.432 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:34.432 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:34.486 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:53:34.486 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:34.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-06T07:53:34.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-06T07:53:34.509 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:34.509 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:34.556 INFO:tasks.ceph:Setting up mon nodes... 2024-04-06T07:53:34.557 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-06T07:53:34.557 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-06T07:53:34.604 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-06T07:53:34.607 INFO:tasks.ceph:Setting up mds nodes... 2024-04-06T07:53:34.607 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-06T07:53:34.607 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-04-06T07:53:34.683 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-04-06T07:53:34.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-06T07:53:34.729 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-06T07:53:34.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-06T07:53:34.818 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-06T07:53:34.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-06T07:53:34.903 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-06T07:53:34.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-06T07:53:34.985 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-06T07:53:34.996 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-06T07:53:35.077 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-06T07:53:35.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-06T07:53:35.158 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-06T07:53:35.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-06T07:53:35.244 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-06T07:53:35.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-06T07:53:35.331 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-06T07:53:35.340 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-06T07:53:35.341 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-06T07:53:35.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-06T07:53:35.395 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:35.395 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-06T07:53:35.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:53:35.395 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T07:53:35.462 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T07:53:35.463 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T07:53:35.463 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T07:53:35.950 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-04-06T07:53:35.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-04-06T07:53:35.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-06T07:53:36.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-06T07:53:36.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T07:53:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:53:36.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-06T07:53:36.167 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:36.167 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-06T07:53:36.167 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:53:36.167 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-06T07:53:36.227 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T07:53:36.228 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T07:53:36.672 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-04-06T07:53:36.674 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-04-06T07:53:36.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-06T07:53:36.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-06T07:53:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:53:36.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T07:53:36.781 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-06T07:53:36.835 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-06T07:53:36.835 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-06T07:53:36.835 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:53:36.835 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-06T07:53:36.895 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-06T07:53:36.895 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-06T07:53:36.895 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-06T07:53:36.896 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-06T07:53:37.359 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-04-06T07:53:37.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-04-06T07:53:37.361 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-06T07:53:37.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-06T07:53:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: /sbin/restorecon: command not found 2024-04-06T07:53:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:53:37.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:37.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:37.595+0000 7f17308fa640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-06T07:53:37.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:37.599+0000 7f17308fa640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-06T07:53:37.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:37.599+0000 7f17308fa640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-06T07:53:37.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:37.599+0000 7f17308fa640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-06T07:53:37.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:37.599+0000 7f17308fa640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-06T07:53:37.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:37.607+0000 7f17308fa640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-06T07:53:40.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-06T07:53:40.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:40.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:40.675+0000 7f992e0bb640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-06T07:53:40.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:40.675+0000 7f992e0bb640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-06T07:53:40.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:40.675+0000 7f992e0bb640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-06T07:53:40.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:40.675+0000 7f992e0bb640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-06T07:53:40.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:40.675+0000 7f992e0bb640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-06T07:53:40.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:40.675+0000 7f992e0bb640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-06T07:53:43.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-06T07:53:43.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:43.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:43.803+0000 7f5baeb2e640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-06T07:53:43.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:43.803+0000 7f5baeb2e640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-06T07:53:43.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:43.803+0000 7f5baeb2e640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-06T07:53:43.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:43.803+0000 7f5baeb2e640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-06T07:53:43.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:43.803+0000 7f5baeb2e640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-06T07:53:43.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T07:53:43.803+0000 7f5baeb2e640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-06T07:53:46.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-06T07:53:46.833 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-06T07:53:46.833 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:46.833 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-06T07:53:46.849 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:46.849 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-06T07:53:46.888 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:46.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-06T07:53:46.943 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:46.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-06T07:53:46.999 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-04-06T07:53:46.999 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-04-06T07:53:47.005 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.005 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-06T07:53:47.044 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.044 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-06T07:53:47.092 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.092 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-04-06T07:53:47.140 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.140 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-04-06T07:53:47.188 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.188 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-04-06T07:53:47.236 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.236 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-04-06T07:53:47.284 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.284 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-04-06T07:53:47.332 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:47.332 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-04-06T07:53:47.383 INFO:tasks.ceph:Adding keys to all mons... 2024-04-06T07:53:47.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-06T07:53:47.434 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB+/xBmZh7nIxAApdDXBFX5EbcrDgFgCe5lVQ== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCB/xBmWAXZIxAA/6K1UVEfC5TBxbEyZ2px+Q== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCE/xBmXWuKKBAA3uX0pL13klxoeVTdM2xRBA== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCH/xBm+nEuMBAApIyH5vFRP5Ouv99t7/6OkA== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB+/xBmXsKYKBAAykqLNbDVCac5fOIt2q2CQQ== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB+/xBmrR1uKxAAXtSA2hlXlhkCuSX89KoIwA== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.0] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB+/xBmg9KuMBAAyZq+qXdkrvteD3z6v251EQ== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.1] 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB+/xBmP8fKNRAAplJChH6CW1sWpFC3QnCo+g== 2024-04-06T07:53:47.435 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.2] 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB+/xBmDaawOhAAIoeZZZai5+euVqWyAk+zDQ== 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.3] 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB//xBmmmeEBBAAbk5fpOipX3unzKuYz4LP5g== 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.4] 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB//xBmgspmCRAAd5/HZLWpflUem6BCWtfsgQ== 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.5] 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB//xBmur6EDhAAWflKnSfKgPNYYNXS2ioFTg== 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout:[client.mirror.6] 2024-04-06T07:53:47.436 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB//xBm94epExAA2awKnJaCVNmU3WfAnbTGtg== 2024-04-06T07:53:47.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-06T07:53:47.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T07:53:47.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T07:53:47.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-06T07:53:47.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:47.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:47.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:47.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:48.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:48.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:48.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:48.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:48.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-06T07:53:48.423 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-06T07:53:48.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-06T07:53:48.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-06T07:53:48.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-06T07:53:48.580 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-06T07:53:48.630 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-06T07:53:48.631 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-06T07:53:48.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-06T07:53:48.673 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-06T07:53:48.673 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-06T07:53:48.673 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-06T07:53:48.674 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-06T07:53:48.676 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-06T07:53:48.676 INFO:tasks.ceph:Setting crush tunables to default 2024-04-06T07:53:48.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-06T07:53:48.927 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-04-06T07:53:48.932 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:48.925+0000 7fe5a8b9d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T07:53:48.945 INFO:tasks.ceph:check_enable_crimson: False 2024-04-06T07:53:48.945 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-06T07:53:48.945 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:48.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-06T07:53:48.957 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:48.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-06T07:53:49.011 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:53:49.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-06T07:53:49.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster cluster2 osd new 420cbb52-6f75-4ab9-9d31-325afe8c92d2 0 2024-04-06T07:53:49.307 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-06T07:53:49.312 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:49.305+0000 7fe5a8b9d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T07:53:49.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster cluster2 osd new 737bc739-4d56-47b2-83b9-d2cff9bf4416 1 2024-04-06T07:53:49.445 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:49.437+0000 7fe5a8b9d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T07:53:49.526 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-04-06T07:53:49.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster cluster2 osd new 340934bd-b0c5-4357-8724-8ca9eda2a7e8 2 2024-04-06T07:53:49.739 INFO:teuthology.orchestra.run.smithi129.stdout:2 2024-04-06T07:53:49.756 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-06T07:53:49.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-06T07:53:49.758 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-06T07:53:49.758 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-06T07:53:49.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-06T07:53:49.760 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-06T07:53:49.760 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-06T07:53:49.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-06T07:53:49.762 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-06T07:53:49.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T07:53:49.878 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:49.873+0000 7fe5a8b9d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T07:53:49.931 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:49.925+0000 7fe5a8b9d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T07:53:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:53:49.940 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","created":"2024-04-06T07:53:48.806348+0000","modified":"2024-04-06T07:53:49.735755+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T07:53:49.735755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"420cbb52-6f75-4ab9-9d31-325afe8c92d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"737bc739-4d56-47b2-83b9-d2cff9bf4416","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"340934bd-b0c5-4357-8724-8ca9eda2a7e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:49.957 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-06T07:53:49.957 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-06T07:53:49.980 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:49.973+0000 7fe5a8b9d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T07:53:50.058 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.053+0000 7fe5a8b9d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T07:53:50.109 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.101+0000 7fe5a8b9d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T07:53:50.184 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.177+0000 7fe5a8b9d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T07:53:50.237 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.229+0000 7fe5a8b9d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T07:53:50.259 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-06T07:53:50.365 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T07:53:50.359+0000 7fcd41d64640 -1 Falling back to public interface 2024-04-06T07:53:50.369 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T07:53:50.363+0000 7fc421fa6640 -1 Falling back to public interface 2024-04-06T07:53:50.376 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T07:53:50.371+0000 7f5a25738640 -1 Falling back to public interface 2024-04-06T07:53:50.380 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.373+0000 7fe5a8b9d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T07:53:50.433 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.425+0000 7fe5a8b9d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T07:53:50.445 INFO:teuthology.misc.health.smithi129.stdout: 2024-04-06T07:53:50.445 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":5,"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","created":"2024-04-06T07:53:48.806348+0000","modified":"2024-04-06T07:53:49.735755+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T07:53:49.735755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"420cbb52-6f75-4ab9-9d31-325afe8c92d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"737bc739-4d56-47b2-83b9-d2cff9bf4416","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"340934bd-b0c5-4357-8724-8ca9eda2a7e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:50.461 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T07:53:50.656 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-06T07:53:50.657 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-06T07:53:50.657 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr: from numpy import show_config as show_numpy_config 2024-04-06T07:53:50.664 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.657+0000 7fe5a8b9d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T07:53:50.801 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.793+0000 7fe5a8b9d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T07:53:50.913 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.905+0000 7fe5a8b9d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T07:53:50.977 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:50.969+0000 7fe5a8b9d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T07:53:51.113 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:51.109+0000 7fe5a8b9d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T07:53:51.608 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:51.601+0000 7fe5a8b9d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T07:53:51.684 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:51.677+0000 7fe5a8b9d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T07:53:51.832 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:51.825+0000 7fe5a8b9d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T07:53:51.908 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:51.901+0000 7fe5a8b9d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T07:53:52.200 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:52.193+0000 7fe5a8b9d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T07:53:52.256 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:52.249+0000 7fe5a8b9d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T07:53:53.288 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T07:53:53.280+0000 7fe550252640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-06T07:53:53.657 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T07:53:53.651+0000 7f5a25738640 -1 osd.2 0 log_to_monitors true 2024-04-06T07:53:54.069 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T07:53:54.063+0000 7fcd41d64640 -1 osd.1 0 log_to_monitors true 2024-04-06T07:53:54.167 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T07:53:54.159+0000 7fc421fa6640 -1 osd.0 0 log_to_monitors true 2024-04-06T07:53:56.764 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-06T07:53:57.237 INFO:teuthology.misc.health.smithi129.stdout: 2024-04-06T07:53:57.237 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":7,"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","created":"2024-04-06T07:53:48.806348+0000","modified":"2024-04-06T07:53:55.337379+0000","last_up_change":"0.000000","last_in_change":"2024-04-06T07:53:49.735755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"420cbb52-6f75-4ab9-9d31-325afe8c92d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"737bc739-4d56-47b2-83b9-d2cff9bf4416","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"340934bd-b0c5-4357-8724-8ca9eda2a7e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:53:57.240 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T07:53:57.235+0000 7f5a216dd640 -1 osd.2 0 waiting for initial osdmap 2024-04-06T07:53:57.245 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T07:53:57.239+0000 7fc41df4b640 -1 osd.0 0 waiting for initial osdmap 2024-04-06T07:53:57.259 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-06T07:53:57.280 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T07:53:57.275+0000 7fcd3e520640 -1 osd.1 0 waiting for initial osdmap 2024-04-06T07:53:57.288 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T07:53:57.283+0000 7fc415d7e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T07:53:57.292 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T07:53:57.287+0000 7f5a19510640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T07:53:57.300 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T07:53:57.295+0000 7fcd35b3c640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-06T07:54:03.563 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-06T07:54:03.858 INFO:teuthology.misc.health.smithi129.stdout: 2024-04-06T07:54:03.859 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":11,"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","created":"2024-04-06T07:53:48.806348+0000","modified":"2024-04-06T07:54:01.292497+0000","last_up_change":"2024-04-06T07:53:58.280069+0000","last_in_change":"2024-04-06T07:53:49.735755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:58.355249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"420cbb52-6f75-4ab9-9d31-325afe8c92d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":24003},{"type":"v1","addr":"172.21.15.129:6809","nonce":24003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":24003},{"type":"v1","addr":"172.21.15.129:6811","nonce":24003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":24003},{"type":"v1","addr":"172.21.15.129:6815","nonce":24003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":24003},{"type":"v1","addr":"172.21.15.129:6813","nonce":24003}]},"public_addr":"172.21.15.129:6809/24003","cluster_addr":"172.21.15.129:6811/24003","heartbeat_back_addr":"172.21.15.129:6815/24003","heartbeat_front_addr":"172.21.15.129:6813/24003","state":["exists","up"]},{"osd":1,"uuid":"737bc739-4d56-47b2-83b9-d2cff9bf4416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":24004},{"type":"v1","addr":"172.21.15.129:6801","nonce":24004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":24004},{"type":"v1","addr":"172.21.15.129:6803","nonce":24004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":24004},{"type":"v1","addr":"172.21.15.129:6807","nonce":24004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":24004},{"type":"v1","addr":"172.21.15.129:6805","nonce":24004}]},"public_addr":"172.21.15.129:6801/24004","cluster_addr":"172.21.15.129:6803/24004","heartbeat_back_addr":"172.21.15.129:6807/24004","heartbeat_front_addr":"172.21.15.129:6805/24004","state":["exists","up"]},{"osd":2,"uuid":"340934bd-b0c5-4357-8724-8ca9eda2a7e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":24005},{"type":"v1","addr":"172.21.15.129:6817","nonce":24005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":24005},{"type":"v1","addr":"172.21.15.129:6819","nonce":24005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":24005},{"type":"v1","addr":"172.21.15.129:6823","nonce":24005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":24005},{"type":"v1","addr":"172.21.15.129:6821","nonce":24005}]},"public_addr":"172.21.15.129:6817/24005","cluster_addr":"172.21.15.129:6819/24005","heartbeat_back_addr":"172.21.15.129:6823/24005","heartbeat_front_addr":"172.21.15.129:6821/24005","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:55.153209+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:55.118199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:54.628270+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:54:03.876 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-06T07:54:03.877 INFO:tasks.ceph:Creating RBD pool 2024-04-06T07:54:03.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-06T07:54:04.766 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'rbd' created 2024-04-06T07:54:04.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-06T07:54:05.775 INFO:teuthology.orchestra.run.smithi129.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-06T07:54:05.806 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-06T07:54:05.806 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-06T07:54:05.807 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-06T07:54:05.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-06T07:54:05.807 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-06T07:54:06.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:54:06.158 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":28016},{"type":"v1","addr":"172.21.15.47:6827","nonce":28016}]},"active_addr":"172.21.15.47:6827/28016","active_change":"2024-04-06T07:53:52.275019+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":4056365043}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":2350724750}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":2165929034}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.47:0","nonce":3735407947}]}]} 2024-04-06T07:54:06.164 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-06T07:54:06.164 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-06T07:54:06.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T07:54:06.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:54:06.459 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":13,"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","created":"2024-04-06T07:53:48.806348+0000","modified":"2024-04-06T07:54:05.762763+0000","last_up_change":"2024-04-06T07:53:58.280069+0000","last_in_change":"2024-04-06T07:53:49.735755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:58.355249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T07:54:04.208377+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"420cbb52-6f75-4ab9-9d31-325afe8c92d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":24003},{"type":"v1","addr":"172.21.15.129:6809","nonce":24003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":24003},{"type":"v1","addr":"172.21.15.129:6811","nonce":24003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":24003},{"type":"v1","addr":"172.21.15.129:6815","nonce":24003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":24003},{"type":"v1","addr":"172.21.15.129:6813","nonce":24003}]},"public_addr":"172.21.15.129:6809/24003","cluster_addr":"172.21.15.129:6811/24003","heartbeat_back_addr":"172.21.15.129:6815/24003","heartbeat_front_addr":"172.21.15.129:6813/24003","state":["exists","up"]},{"osd":1,"uuid":"737bc739-4d56-47b2-83b9-d2cff9bf4416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":24004},{"type":"v1","addr":"172.21.15.129:6801","nonce":24004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":24004},{"type":"v1","addr":"172.21.15.129:6803","nonce":24004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":24004},{"type":"v1","addr":"172.21.15.129:6807","nonce":24004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":24004},{"type":"v1","addr":"172.21.15.129:6805","nonce":24004}]},"public_addr":"172.21.15.129:6801/24004","cluster_addr":"172.21.15.129:6803/24004","heartbeat_back_addr":"172.21.15.129:6807/24004","heartbeat_front_addr":"172.21.15.129:6805/24004","state":["exists","up"]},{"osd":2,"uuid":"340934bd-b0c5-4357-8724-8ca9eda2a7e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":24005},{"type":"v1","addr":"172.21.15.129:6817","nonce":24005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":24005},{"type":"v1","addr":"172.21.15.129:6819","nonce":24005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":24005},{"type":"v1","addr":"172.21.15.129:6823","nonce":24005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":24005},{"type":"v1","addr":"172.21.15.129:6821","nonce":24005}]},"public_addr":"172.21.15.129:6817/24005","cluster_addr":"172.21.15.129:6819/24005","heartbeat_back_addr":"172.21.15.129:6823/24005","heartbeat_front_addr":"172.21.15.129:6821/24005","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:55.153209+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:55.118199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:54.628270+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:54:06.478 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-06T07:54:06.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-06T07:54:06.777 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:54:06.777 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":13,"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","created":"2024-04-06T07:53:48.806348+0000","modified":"2024-04-06T07:54:05.762763+0000","last_up_change":"2024-04-06T07:53:58.280069+0000","last_in_change":"2024-04-06T07:53:49.735755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:58.355249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T07:54:04.208377+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"420cbb52-6f75-4ab9-9d31-325afe8c92d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":24003},{"type":"v1","addr":"172.21.15.129:6809","nonce":24003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":24003},{"type":"v1","addr":"172.21.15.129:6811","nonce":24003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":24003},{"type":"v1","addr":"172.21.15.129:6815","nonce":24003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":24003},{"type":"v1","addr":"172.21.15.129:6813","nonce":24003}]},"public_addr":"172.21.15.129:6809/24003","cluster_addr":"172.21.15.129:6811/24003","heartbeat_back_addr":"172.21.15.129:6815/24003","heartbeat_front_addr":"172.21.15.129:6813/24003","state":["exists","up"]},{"osd":1,"uuid":"737bc739-4d56-47b2-83b9-d2cff9bf4416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":24004},{"type":"v1","addr":"172.21.15.129:6801","nonce":24004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":24004},{"type":"v1","addr":"172.21.15.129:6803","nonce":24004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":24004},{"type":"v1","addr":"172.21.15.129:6807","nonce":24004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":24004},{"type":"v1","addr":"172.21.15.129:6805","nonce":24004}]},"public_addr":"172.21.15.129:6801/24004","cluster_addr":"172.21.15.129:6803/24004","heartbeat_back_addr":"172.21.15.129:6807/24004","heartbeat_front_addr":"172.21.15.129:6805/24004","state":["exists","up"]},{"osd":2,"uuid":"340934bd-b0c5-4357-8724-8ca9eda2a7e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":24005},{"type":"v1","addr":"172.21.15.129:6817","nonce":24005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":24005},{"type":"v1","addr":"172.21.15.129:6819","nonce":24005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":24005},{"type":"v1","addr":"172.21.15.129:6823","nonce":24005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":24005},{"type":"v1","addr":"172.21.15.129:6821","nonce":24005}]},"public_addr":"172.21.15.129:6817/24005","cluster_addr":"172.21.15.129:6819/24005","heartbeat_back_addr":"172.21.15.129:6823/24005","heartbeat_front_addr":"172.21.15.129:6821/24005","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:55.153209+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:55.118199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:54.628270+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T07:54:06.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-06T07:54:06.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-06T07:54:06.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-06T07:54:06.945 INFO:teuthology.orchestra.run.smithi129.stdout:34359738374 2024-04-06T07:54:06.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-06T07:54:06.963 INFO:teuthology.orchestra.run.smithi129.stdout:34359738374 2024-04-06T07:54:06.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-06T07:54:06.966 INFO:teuthology.orchestra.run.smithi129.stdout:34359738374 2024-04-06T07:54:06.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-06T07:54:07.246 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-06T07:54:07.263 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-04-06T07:54:07.280 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-06T07:54:07.296 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-04-06T07:54:07.297 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-06T07:54:07.314 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-04-06T07:54:08.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-06T07:54:08.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-06T07:54:08.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-06T07:54:08.571 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-06T07:54:08.588 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-04-06T07:54:08.610 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-06T07:54:08.627 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-04-06T07:54:08.635 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-06T07:54:08.651 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-04-06T07:54:09.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-06T07:54:09.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-06T07:54:09.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-06T07:54:09.895 INFO:teuthology.orchestra.run.smithi129.stdout:34359738374 2024-04-06T07:54:09.919 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2024-04-06T07:54:09.919 DEBUG:teuthology.parallel:result is None 2024-04-06T07:54:09.923 INFO:teuthology.orchestra.run.smithi129.stdout:34359738374 2024-04-06T07:54:09.940 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-04-06T07:54:09.940 DEBUG:teuthology.parallel:result is None 2024-04-06T07:54:09.974 INFO:teuthology.orchestra.run.smithi129.stdout:34359738374 2024-04-06T07:54:09.990 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2024-04-06T07:54:09.990 DEBUG:teuthology.parallel:result is None 2024-04-06T07:54:09.991 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-06T07:54:09.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T07:54:10.306 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:54:10.306 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T07:54:10.322 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T07:54:10.286197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82580,"kb_used_data":1512,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032940,"statfs":{"total":289910292480,"available":289825730560,"internally_reserved":0,"allocated":1548288,"data_stored":1281649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001885"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942439+0000","last_change":"2024-04-06T07:54:05.782357+0000","last_active":"2024-04-06T07:54:06.942439+0000","last_peered":"2024-04-06T07:54:06.942439+0000","last_clean":"2024-04-06T07:54:06.942439+0000","last_became_active":"2024-04-06T07:54:05.781667+0000","last_became_peered":"2024-04-06T07:54:05.781667+0000","last_unstale":"2024-04-06T07:54:06.942439+0000","last_undegraded":"2024-04-06T07:54:06.942439+0000","last_fullsized":"2024-04-06T07:54:06.942439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:30.495409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942422+0000","last_change":"2024-04-06T07:54:05.780956+0000","last_active":"2024-04-06T07:54:06.942422+0000","last_peered":"2024-04-06T07:54:06.942422+0000","last_clean":"2024-04-06T07:54:06.942422+0000","last_became_active":"2024-04-06T07:54:05.780561+0000","last_became_peered":"2024-04-06T07:54:05.780561+0000","last_unstale":"2024-04-06T07:54:06.942422+0000","last_undegraded":"2024-04-06T07:54:06.942422+0000","last_fullsized":"2024-04-06T07:54:06.942422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.353333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942399+0000","last_change":"2024-04-06T07:54:05.783964+0000","last_active":"2024-04-06T07:54:06.942399+0000","last_peered":"2024-04-06T07:54:06.942399+0000","last_clean":"2024-04-06T07:54:06.942399+0000","last_became_active":"2024-04-06T07:54:05.781296+0000","last_became_peered":"2024-04-06T07:54:05.781296+0000","last_unstale":"2024-04-06T07:54:06.942399+0000","last_undegraded":"2024-04-06T07:54:06.942399+0000","last_fullsized":"2024-04-06T07:54:06.942399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:43.301937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942383+0000","last_change":"2024-04-06T07:54:05.782925+0000","last_active":"2024-04-06T07:54:06.942383+0000","last_peered":"2024-04-06T07:54:06.942383+0000","last_clean":"2024-04-06T07:54:06.942383+0000","last_became_active":"2024-04-06T07:54:05.782116+0000","last_became_peered":"2024-04-06T07:54:05.782116+0000","last_unstale":"2024-04-06T07:54:06.942383+0000","last_undegraded":"2024-04-06T07:54:06.942383+0000","last_fullsized":"2024-04-06T07:54:06.942383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:18.319096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.781272+0000","last_change":"2024-04-06T07:54:05.783983+0000","last_active":"2024-04-06T07:54:06.781272+0000","last_peered":"2024-04-06T07:54:06.781272+0000","last_clean":"2024-04-06T07:54:06.781272+0000","last_became_active":"2024-04-06T07:54:05.783370+0000","last_became_peered":"2024-04-06T07:54:05.783370+0000","last_unstale":"2024-04-06T07:54:06.781272+0000","last_undegraded":"2024-04-06T07:54:06.781272+0000","last_fullsized":"2024-04-06T07:54:06.781272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:14.816075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.929872+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T07:54:06.929872+0000","last_peered":"2024-04-06T07:54:06.929872+0000","last_clean":"2024-04-06T07:54:06.929872+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T07:54:06.929872+0000","last_undegraded":"2024-04-06T07:54:06.929872+0000","last_fullsized":"2024-04-06T07:54:06.929872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.929857+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T07:54:06.929857+0000","last_peered":"2024-04-06T07:54:06.929857+0000","last_clean":"2024-04-06T07:54:06.929857+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T07:54:06.929857+0000","last_undegraded":"2024-04-06T07:54:06.929857+0000","last_fullsized":"2024-04-06T07:54:06.929857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942358+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T07:54:06.942358+0000","last_peered":"2024-04-06T07:54:06.942358+0000","last_clean":"2024-04-06T07:54:06.942358+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T07:54:06.942358+0000","last_undegraded":"2024-04-06T07:54:06.942358+0000","last_fullsized":"2024-04-06T07:54:06.942358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:05.780060+0000","last_change":"2024-04-06T07:54:00.377140+0000","last_active":"2024-04-06T07:54:05.780060+0000","last_peered":"2024-04-06T07:54:05.780060+0000","last_clean":"2024-04-06T07:54:05.780060+0000","last_became_active":"2024-04-06T07:54:00.376155+0000","last_became_peered":"2024-04-06T07:54:00.376155+0000","last_unstale":"2024-04-06T07:54:05.780060+0000","last_undegraded":"2024-04-06T07:54:05.780060+0000","last_fullsized":"2024-04-06T07:54:05.780060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:08:34.216215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T07:54:10.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T07:54:10.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:54:10.592 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T07:54:10.608 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-06T07:54:10.286197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82580,"kb_used_data":1512,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032940,"statfs":{"total":289910292480,"available":289825730560,"internally_reserved":0,"allocated":1548288,"data_stored":1281649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001885"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942439+0000","last_change":"2024-04-06T07:54:05.782357+0000","last_active":"2024-04-06T07:54:06.942439+0000","last_peered":"2024-04-06T07:54:06.942439+0000","last_clean":"2024-04-06T07:54:06.942439+0000","last_became_active":"2024-04-06T07:54:05.781667+0000","last_became_peered":"2024-04-06T07:54:05.781667+0000","last_unstale":"2024-04-06T07:54:06.942439+0000","last_undegraded":"2024-04-06T07:54:06.942439+0000","last_fullsized":"2024-04-06T07:54:06.942439+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T10:50:30.495409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942422+0000","last_change":"2024-04-06T07:54:05.780956+0000","last_active":"2024-04-06T07:54:06.942422+0000","last_peered":"2024-04-06T07:54:06.942422+0000","last_clean":"2024-04-06T07:54:06.942422+0000","last_became_active":"2024-04-06T07:54:05.780561+0000","last_became_peered":"2024-04-06T07:54:05.780561+0000","last_unstale":"2024-04-06T07:54:06.942422+0000","last_undegraded":"2024-04-06T07:54:06.942422+0000","last_fullsized":"2024-04-06T07:54:06.942422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:08:18.353333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942399+0000","last_change":"2024-04-06T07:54:05.783964+0000","last_active":"2024-04-06T07:54:06.942399+0000","last_peered":"2024-04-06T07:54:06.942399+0000","last_clean":"2024-04-06T07:54:06.942399+0000","last_became_active":"2024-04-06T07:54:05.781296+0000","last_became_peered":"2024-04-06T07:54:05.781296+0000","last_unstale":"2024-04-06T07:54:06.942399+0000","last_undegraded":"2024-04-06T07:54:06.942399+0000","last_fullsized":"2024-04-06T07:54:06.942399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:43:43.301937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942383+0000","last_change":"2024-04-06T07:54:05.782925+0000","last_active":"2024-04-06T07:54:06.942383+0000","last_peered":"2024-04-06T07:54:06.942383+0000","last_clean":"2024-04-06T07:54:06.942383+0000","last_became_active":"2024-04-06T07:54:05.782116+0000","last_became_peered":"2024-04-06T07:54:05.782116+0000","last_unstale":"2024-04-06T07:54:06.942383+0000","last_undegraded":"2024-04-06T07:54:06.942383+0000","last_fullsized":"2024-04-06T07:54:06.942383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:21:18.319096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.781272+0000","last_change":"2024-04-06T07:54:05.783983+0000","last_active":"2024-04-06T07:54:06.781272+0000","last_peered":"2024-04-06T07:54:06.781272+0000","last_clean":"2024-04-06T07:54:06.781272+0000","last_became_active":"2024-04-06T07:54:05.783370+0000","last_became_peered":"2024-04-06T07:54:05.783370+0000","last_unstale":"2024-04-06T07:54:06.781272+0000","last_undegraded":"2024-04-06T07:54:06.781272+0000","last_fullsized":"2024-04-06T07:54:06.781272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:32:14.816075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.929872+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T07:54:06.929872+0000","last_peered":"2024-04-06T07:54:06.929872+0000","last_clean":"2024-04-06T07:54:06.929872+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T07:54:06.929872+0000","last_undegraded":"2024-04-06T07:54:06.929872+0000","last_fullsized":"2024-04-06T07:54:06.929872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.929857+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T07:54:06.929857+0000","last_peered":"2024-04-06T07:54:06.929857+0000","last_clean":"2024-04-06T07:54:06.929857+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T07:54:06.929857+0000","last_undegraded":"2024-04-06T07:54:06.929857+0000","last_fullsized":"2024-04-06T07:54:06.929857+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:06.942358+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T07:54:06.942358+0000","last_peered":"2024-04-06T07:54:06.942358+0000","last_clean":"2024-04-06T07:54:06.942358+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T07:54:06.942358+0000","last_undegraded":"2024-04-06T07:54:06.942358+0000","last_fullsized":"2024-04-06T07:54:06.942358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-06T07:54:05.780060+0000","last_change":"2024-04-06T07:54:00.377140+0000","last_active":"2024-04-06T07:54:05.780060+0000","last_peered":"2024-04-06T07:54:05.780060+0000","last_clean":"2024-04-06T07:54:05.780060+0000","last_became_active":"2024-04-06T07:54:00.376155+0000","last_became_peered":"2024-04-06T07:54:00.376155+0000","last_unstale":"2024-04-06T07:54:05.780060+0000","last_undegraded":"2024-04-06T07:54:05.780060+0000","last_fullsized":"2024-04-06T07:54:05.780060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:08:34.216215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T07:54:10.609 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-06T07:54:10.609 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-06T07:54:10.609 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-06T07:54:10.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-06T07:54:10.921 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T07:54:10.921 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T07:54:10.939 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-06T07:54:10.940 INFO:teuthology.run_tasks:Running task exec... 2024-04-06T07:54:10.956 INFO:teuthology.task.exec:Executing custom commands... 2024-04-06T07:54:10.956 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T07:54:10.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror 2024-04-06T07:54:11.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.0 2024-04-06T07:54:11.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:12.094 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.1 2024-04-06T07:54:12.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:12.473 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.2 2024-04-06T07:54:12.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:12.852 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.3 2024-04-06T07:54:12.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:13.212 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror 2024-04-06T07:54:13.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.0 2024-04-06T07:54:13.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:13.969 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.1 2024-04-06T07:54:13.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:14.329 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.2 2024-04-06T07:54:14.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-06T07:54:14.686 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.mirror.3 2024-04-06T07:54:14.704 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T07:54:14.729 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T07:54:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T07:54:14.731 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T07:54:14.740 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T07:54:14.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T07:54:14.742 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T07:54:14.750 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T07:54:14.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T07:54:14.753 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T07:54:14.761 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T07:54:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T07:54:14.764 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T07:54:14.773 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T07:54:14.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T07:54:14.775 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T07:54:14.785 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T07:54:14.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T07:54:14.787 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T07:54:14.796 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T07:54:14.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T07:54:14.799 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-06T07:54:14.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T07:54:14.809 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T07:54:14.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T07:54:14.812 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-06T07:54:14.828 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1712390054 2024-04-06T07:54:14.829 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-06T07:54:14.829 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-06T07:54:14.878 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1712390054 2024-04-06T07:54:14.878 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-06T07:54:14.878 INFO:teuthology.run_tasks:Running task workunit... 2024-04-06T07:54:14.891 INFO:tasks.workunit:Pulling workunits from ref 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T07:54:14.892 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-06T07:54:14.892 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T07:54:14.893 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-04-06T07:54:14.893 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-04-06T07:54:14.893 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-04-06T07:54:14.893 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-04-06T07:54:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T07:54:14.899 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-06T07:54:14.899 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T07:54:14.947 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T07:54:14.947 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-06T07:54:14.995 INFO:tasks.workunit:timeout=6h 2024-04-06T07:54:14.995 INFO:tasks.workunit:cleanup=True 2024-04-06T07:54:14.996 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 052adcadd0534eddb0d73c5afbc083a84cd9dc46 2024-04-06T07:54:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-06T07:55:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Note: switching to '052adcadd0534eddb0d73c5afbc083a84cd9dc46'. 2024-04-06T07:55:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-06T07:55:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:state without impacting any branches by switching back to a branch. 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: git switch -c 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Or undo this operation with: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: git switch - 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-06T07:55:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:55:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:HEAD is now at 052adcadd05 Merge PR #56567 into reef 2024-04-06T07:55:21.868 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-06T07:55:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-06T07:55:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-06T07:55:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-06T07:55:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-06T07:55:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-06T07:55:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-06T07:55:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-06T07:55:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-06T07:55:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-06T07:55:23.114 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-06T07:55:23.114 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-06T07:55:23.159 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-04-06T07:55:23.161 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-04-06T07:55:23.161 DEBUG:teuthology.orchestra.run.smithi129:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=052adcadd0534eddb0d73c5afbc083a84cd9dc46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-06T07:55:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ IMAGE_COUNT=50 2024-04-06T07:55:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ export LOCKDEP=0 2024-04-06T07:55:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ LOCKDEP=0 2024-04-06T07:55:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-06T07:55:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-06T07:55:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ type xmlstarlet 2024-04-06T07:55:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ XMLSTARLET=xmlstarlet 2024-04-06T07:55:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ RBD_MIRROR_INSTANCES=4 2024-04-06T07:55:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CLUSTER1=cluster1 2024-04-06T07:55:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CLUSTER2=cluster2 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ POOL=mirror 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ PARENT_POOL=mirror_parent 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ NS1=ns1 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ NS2=ns2 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ TEMPDIR= 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ID=mirror 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ MIRROR_POOL_MODE=image 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS='--id mirror' 2024-04-06T07:55:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ LAST_MIRROR_INSTANCE=3 2024-04-06T07:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-06T07:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-06T07:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-06T07:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_BIN=. 2024-04-06T07:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_SRC=. 2024-04-06T07:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-06T07:55:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 -gt 0 ']' 2024-04-06T07:55:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ setup 2024-04-06T07:55:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local c 2024-04-06T07:55:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-06T07:55:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ setup_tempdir 2024-04-06T07:55:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mktemp -d 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ TEMPDIR=/tmp/tmp.phadx0IwJQ 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z 1 ']' 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ setup_pools cluster1 cluster2 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local remote_cluster=cluster2 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mon_map_file 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mon_addr 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local admin_key_file 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local uuid 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-06T07:55:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror' already exists 2024-04-06T07:55:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-06T07:55:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror_parent' already exists 2024-04-06T07:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-06T07:55:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-06T07:55:28.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.34723222300526113) > weight (0.25) 2024-04-06T07:55:28.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.294053805535447) > weight (0.25) 2024-04-06T07:55:28.894 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T07:55:28.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T07:55:28.895 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T07:55:28.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T07:55:28.893+0000 7f1ecbc8d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 28950) UID: 1000 2024-04-06T07:55:30.895 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T07:55:30.895 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T07:55:30.896 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T07:55:30.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T07:55:30.896 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T07:55:30.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T07:55:30.899 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-04-06T07:55:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-04-06T07:55:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-06T07:55:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-06T07:55:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-06T07:55:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-04-06T07:55:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-06T07:55:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z ']' 2024-04-06T07:55:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z ']' 2024-04-06T07:55:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_add cluster1 mirror cluster2 2024-04-06T07:55:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local client_cluster=cluster2 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local remote_cluster=cluster2 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local uuid_var_name 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local error_code 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local peer_uuid 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:55:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-06T07:55:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_uuid=e12014c6-c75a-4166-beec-da99038f8a61 2024-04-06T07:55:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T07:55:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:55:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T07:55:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 -eq 0 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror_parent 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local client_cluster=cluster2 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local remote_cluster=cluster2 2024-04-06T07:55:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local uuid_var_name 2024-04-06T07:55:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local error_code 2024-04-06T07:55:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local peer_uuid 2024-04-06T07:55:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T07:55:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:55:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-06T07:55:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_uuid=dc527285-eafb-4224-8df9-7ee55e68e8cc 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 -eq 0 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ setup_pools cluster2 cluster1 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local remote_cluster=cluster1 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mon_map_file 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mon_addr 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local admin_key_file 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local uuid 2024-04-06T07:55:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-06T07:55:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror' already exists 2024-04-06T07:55:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-06T07:55:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror_parent' already exists 2024-04-06T07:55:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-06T07:55:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-06T07:55:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T07:55:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-04-06T07:55:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-06T07:55:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-06T07:55:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-06T07:55:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-04-06T07:55:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-06T07:55:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z ']' 2024-04-06T07:55:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z ']' 2024-04-06T07:55:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_add cluster2 mirror cluster1 2024-04-06T07:55:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local client_cluster=cluster1 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local remote_cluster=cluster1 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local uuid_var_name 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local error_code 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local peer_uuid 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:55:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_uuid=c4836cb5-a278-4660-ac19-117d9d4862c5 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 -eq 0 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:55:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror_parent 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local client_cluster=cluster1 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local remote_cluster=cluster1 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local uuid_var_name 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local error_code 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local peer_uuid 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 16 32 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:55:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-06T07:55:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ peer_uuid=cf06de13-b8b7-4d6d-974c-36536eefc5b6 2024-04-06T07:55:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T07:55:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:55:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 17 ']' 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 -eq 0 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirrors cluster1 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 0 3 2024-04-06T07:55:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster1:0 2024-04-06T07:55:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:0 2024-04-06T07:55:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster1:0 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster1 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=0 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster1 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=0 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster1:1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster1:1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=1 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster1:2 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:2 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster1:2 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster1 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=2 2024-04-06T07:55:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster1 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=2 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster1:3 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:3 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-06T07:55:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster1:3 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster1 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=3 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster1 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=3 2024-04-06T07:55:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirrors cluster2 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 0 3 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster2:0 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:0 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster2:0 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster2 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=0 2024-04-06T07:55:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster2 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster2 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=0 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster2:1 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:1 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster2:1 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster2 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=1 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 1 = cluster2:1 ']' 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster2 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster2 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=1 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster2:2 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:2 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-04-06T07:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster2:2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 2 = cluster2:2 ']' 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster2:3 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:3 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster2:3 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster2 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=3 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 3 = cluster2:3 ']' 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster2 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster2 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=3 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T07:55:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-04-06T07:55:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T07:55:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T07:55:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 07:55:40 TEST: add image and test replay after client crashes 2024-04-06T07:55:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 07:55:40 TEST: add image and test replay after client crashes 2024-04-06T07:55:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=test 2024-04-06T07:55:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-04-06T07:55:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror test 512M 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T07:55:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T07:55:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T07:55:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T07:55:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2024-04-06T07:55:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T07:55:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-04-06T07:55:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T07:55:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-04-06T07:55:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T07:55:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10b1866f9141 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10b1866f9141 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T07:55:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:40 2024 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 07:55:40 2024 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:40 2024 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:55:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T07:55:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:55:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:55:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T07:55:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:55:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T07:55:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T07:55:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T07:55:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:55:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T07:55:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:55:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:55:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:55:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:55:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:55:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:55:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T07:55:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T07:55:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T07:55:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T07:55:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T07:55:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T07:55:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T07:55:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T07:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T07:55:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T07:55:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T07:55:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T07:55:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T07:55:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T07:55:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T07:55:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T07:55:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T07:55:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T07:55:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T07:55:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:55:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 1 2024-04-06T07:55:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:55:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:55:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:55:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:55:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:55:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T07:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T07:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T07:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T07:55:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T07:55:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T07:55:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T07:55:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T07:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T07:55:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T07:55:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T07:55:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T07:55:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T07:55:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T07:55:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T07:55:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T07:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T07:55:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:55:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 2 2024-04-06T07:55:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:55:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:55:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:55:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T07:55:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T07:55:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T07:55:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T07:55:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T07:55:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T07:55:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T07:55:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T07:55:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T07:55:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T07:55:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T07:55:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T07:55:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T07:55:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T07:55:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T07:55:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T07:55:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T07:55:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T07:55:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T07:55:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T07:55:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T07:55:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T07:55:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T07:55:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T07:55:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:55:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 4 2024-04-06T07:55:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:55:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:55:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:55:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:55:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:55:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:55:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:55:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:55:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:55:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:55:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T07:55:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T07:55:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T07:55:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T07:55:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T07:55:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T07:55:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T07:55:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:55:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name= 2024-04-06T07:55:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 10 2024-04-06T07:55:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T07:55:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T07:55:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:55:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:55:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:55:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T07:55:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T07:55:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:55:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T07:55:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 16408576 bytes 2024-04-06T07:55:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 69473280 bytes 2024-04-06T07:55:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 139502592 bytes 2024-04-06T07:55:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 210311168 bytes 2024-04-06T07:55:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 301066240 bytes 2024-04-06T07:55:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 378179584 bytes 2024-04-06T07:55:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 412506112 bytes 2024-04-06T07:55:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 430832640 bytes 2024-04-06T07:55:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 462958592 bytes 2024-04-06T07:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 528500736 bytes 2024-04-06T07:55:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 567371776 bytes 2024-04-06T07:55:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 609030144 bytes 2024-04-06T07:55:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 678479872 bytes 2024-04-06T07:55:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 722246656 bytes 2024-04-06T07:55:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 803531776 bytes 2024-04-06T07:55:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 820960256 bytes 2024-04-06T07:55:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 872614912 bytes 2024-04-06T07:55:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 885060608 bytes 2024-04-06T07:55:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 891692032 bytes 2024-04-06T07:55:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 943572992 bytes 2024-04-06T07:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1020238848 bytes 2024-04-06T07:55:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1065645056 bytes 2024-04-06T07:55:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1146211328 bytes 2024-04-06T07:55:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1196334080 bytes 2024-04-06T07:55:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1245972480 bytes 2024-04-06T07:55:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1250692096 bytes 2024-04-06T07:55:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1315551232 bytes 2024-04-06T07:55:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1401132032 bytes 2024-04-06T07:55:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1437423616 bytes 2024-04-06T07:55:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1461556224 bytes 2024-04-06T07:55:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1522808832 bytes 2024-04-06T07:55:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1542086656 bytes 2024-04-06T07:55:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1616041984 bytes 2024-04-06T07:55:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1625675776 bytes 2024-04-06T07:55:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1716269056 bytes 2024-04-06T07:55:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1755871232 bytes 2024-04-06T07:55:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1803698176 bytes 2024-04-06T07:55:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1845442560 bytes 2024-04-06T07:55:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1875067904 bytes 2024-04-06T07:55:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1903469568 bytes 2024-04-06T07:55:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1931448320 bytes 2024-04-06T07:55:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1999599616 bytes 2024-04-06T07:55:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2095592448 bytes 2024-04-06T07:55:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2117074944 bytes 2024-04-06T07:55:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2165976064 bytes 2024-04-06T07:55:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2167482368 bytes 2024-04-06T07:55:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2205296640 bytes 2024-04-06T07:55:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2243710976 bytes 2024-04-06T07:55:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2288848896 bytes 2024-04-06T07:55:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2353132544 bytes 2024-04-06T07:55:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2378763264 bytes 2024-04-06T07:55:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2436462592 bytes 2024-04-06T07:55:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2499217408 bytes 2024-04-06T07:56:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2513175552 bytes 2024-04-06T07:56:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2526050304 bytes 2024-04-06T07:56:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2547903488 bytes 2024-04-06T07:56:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2576357376 bytes 2024-04-06T07:56:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2610817024 bytes 2024-04-06T07:56:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2662751232 bytes 2024-04-06T07:56:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2692464640 bytes 2024-04-06T07:56:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2700138496 bytes 2024-04-06T07:56:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2743706624 bytes 2024-04-06T07:56:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2755812352 bytes 2024-04-06T07:56:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2784151552 bytes 2024-04-06T07:56:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2803575808 bytes 2024-04-06T07:56:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2818531328 bytes 2024-04-06T07:56:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2858314752 bytes 2024-04-06T07:56:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2900882432 bytes 2024-04-06T07:56:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2989517824 bytes 2024-04-06T07:56:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 3033086976 bytes 2024-04-06T07:56:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 3089451008 bytes 2024-04-06T07:56:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 3090081792 bytes 2024-04-06T07:56:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 3110916096 bytes 2024-04-06T07:56:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 3132249088 bytes 2024-04-06T07:56:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 3213246464 bytes 2024-04-06T07:56:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 3243000832 bytes 2024-04-06T07:56:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 3295475712 bytes 2024-04-06T07:56:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 3384066048 bytes 2024-04-06T07:56:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7900 IOs, 3421808640 bytes 2024-04-06T07:56:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:03.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:56:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: ' 2024-04-06T07:56:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:56:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 1 2024-04-06T07:56:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 2 2024-04-06T07:56:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:56:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:56:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:56:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:56:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:56:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: ' 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 1 2024-04-06T07:56:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 4 2024-04-06T07:56:07.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3852116242601681) > weight (0.25) 2024-04-06T07:56:07.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.401390163114293) > weight (0.25) 2024-04-06T07:56:07.894 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T07:56:07.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T07:56:07.895 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T07:56:07.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T07:56:07.892+0000 7f7a983f6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 28939) UID: 1000 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:56:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: ' 2024-04-06T07:56:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:56:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 1 2024-04-06T07:56:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 8 2024-04-06T07:56:11.895 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T07:56:11.895 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T07:56:11.896 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T07:56:11.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T07:56:11.896 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T07:56:11.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T07:56:11.899 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-06T07:56:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390140,"remote_snapshot_timestamp":1712390140,"replay_state":"idle"} 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.0 on smithi129 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:56:14 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T07:56:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T07:56:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:56:15' 2024-04-06T07:56:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:56:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T07:56:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T07:56:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390140,"remote_snapshot_timestamp":1712390140,"replay_state":"idle"} 2024-04-06T07:56:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:56:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:56:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T07:56:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:56:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T07:56:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.0 on smithi129 2024-04-06T07:56:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T07:56:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:56:14' 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 07:56:14' 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 07:56:14' 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T07:56:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK42154262mirror.0smithi12918.2.2-1059-g4d05f4catrueOK42214263mirror.3smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390140,"remote_snapshot_timestamp":1712390140,"replay_state":"idle"}42154262mirror.0smithi1292024-04-06 07:56:14cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 07:56:15' 2024-04-06T07:56:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T07:56:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T07:56:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:56:14' 2024-04-06T07:56:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T07:56:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T07:56:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:56:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T07:56:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:56:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap1 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap1 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1 2024-04-06T07:56:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T07:56:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:56:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:56:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T07:56:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T07:56:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T07:56:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T07:56:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T07:56:33.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.46335223125651503) > weight (0.25) 2024-04-06T07:56:33.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.25877482847963285) > weight (0.25) 2024-04-06T07:56:33.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7124601021271825) > weight (0.25) 2024-04-06T07:56:33.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8563180750099801) > weight (0.25) 2024-04-06T07:56:33.900 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-04-06T07:56:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:56:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:56:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:56:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:56:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:56:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:56:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:56:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:56:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:56:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:56:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:56:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:56:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:56:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:56:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:56:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:56:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T07:56:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T07:56:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T07:56:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T07:56:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T07:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:56:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:56:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T07:56:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T07:56:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:56:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T07:56:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T07:56:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 6 2024-04-06T07:56:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:56:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:56:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T07:56:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:56:34.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T07:56:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=6 2024-04-06T07:56:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:56:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:56:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T07:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:56:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:56:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:56:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:56:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:56:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:56:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:56:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:56:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 6 = '' 2024-04-06T07:56:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:56:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T07:56:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:56:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:56:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:56:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:56:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:56:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:56:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 6 = '' 2024-04-06T07:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:56:35.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:56:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:56:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:56:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:56:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:56:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:56:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:56:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:56:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 6 = '' 2024-04-06T07:56:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:56:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T07:56:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:56:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:56:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:56:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=6 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 6 = 6 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1 2024-04-06T07:56:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:56:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T07:56:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:56:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T07:56:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:56:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T07:56:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:56:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap1 2024-04-06T07:56:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1 2024-04-06T07:56:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T07:56:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:56:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:56:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:56:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:56:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:56:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:56:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:56:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:56:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap1-clean 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1-clean 2024-04-06T07:56:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T07:56:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T07:56:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T07:56:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T07:56:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T07:56:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:56:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T07:56:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 71688192 bytes 2024-04-06T07:56:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 98101248 bytes 2024-04-06T07:56:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 187427840 bytes 2024-04-06T07:56:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 188032000 bytes 2024-04-06T07:56:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 273625088 bytes 2024-04-06T07:56:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 317049856 bytes 2024-04-06T07:56:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 352715776 bytes 2024-04-06T07:56:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 389509120 bytes 2024-04-06T07:56:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 473850880 bytes 2024-04-06T07:56:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 515900416 bytes 2024-04-06T07:56:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 608238592 bytes 2024-04-06T07:56:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 633774080 bytes 2024-04-06T07:56:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 645900288 bytes 2024-04-06T07:56:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 685890560 bytes 2024-04-06T07:56:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 764662784 bytes 2024-04-06T07:56:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 796150784 bytes 2024-04-06T07:56:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 798600192 bytes 2024-04-06T07:56:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 870361088 bytes 2024-04-06T07:56:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 870985728 bytes 2024-04-06T07:56:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 952786944 bytes 2024-04-06T07:56:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 986721280 bytes 2024-04-06T07:56:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1022465024 bytes 2024-04-06T07:56:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1083867136 bytes 2024-04-06T07:56:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1104052224 bytes 2024-04-06T07:56:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1156603904 bytes 2024-04-06T07:56:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1194433536 bytes 2024-04-06T07:56:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1250725888 bytes 2024-04-06T07:56:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1281244160 bytes 2024-04-06T07:56:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1342671872 bytes 2024-04-06T07:56:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1368200192 bytes 2024-04-06T07:56:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1368818688 bytes 2024-04-06T07:56:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1392140288 bytes 2024-04-06T07:56:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1464080384 bytes 2024-04-06T07:56:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1558685696 bytes 2024-04-06T07:56:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1572702208 bytes 2024-04-06T07:56:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1611907072 bytes 2024-04-06T07:56:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1625094144 bytes 2024-04-06T07:56:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1675964416 bytes 2024-04-06T07:56:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1692702720 bytes 2024-04-06T07:56:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1726821376 bytes 2024-04-06T07:56:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1759301632 bytes 2024-04-06T07:56:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1788949504 bytes 2024-04-06T07:56:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1836526592 bytes 2024-04-06T07:56:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1837979648 bytes 2024-04-06T07:56:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1937854464 bytes 2024-04-06T07:56:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2014686208 bytes 2024-04-06T07:56:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2059203584 bytes 2024-04-06T07:56:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2083390464 bytes 2024-04-06T07:56:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2139630592 bytes 2024-04-06T07:56:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2140272640 bytes 2024-04-06T07:56:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2249239552 bytes 2024-04-06T07:56:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2274999296 bytes 2024-04-06T07:56:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2320343040 bytes 2024-04-06T07:56:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2396798976 bytes 2024-04-06T07:56:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2428474368 bytes 2024-04-06T07:56:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2434975744 bytes 2024-04-06T07:56:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2488522752 bytes 2024-04-06T07:56:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2510142464 bytes 2024-04-06T07:56:51.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2551081984 bytes 2024-04-06T07:56:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2621318144 bytes 2024-04-06T07:56:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2685797376 bytes 2024-04-06T07:56:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2709666816 bytes 2024-04-06T07:56:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2716205056 bytes 2024-04-06T07:56:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2733245440 bytes 2024-04-06T07:56:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2801708032 bytes 2024-04-06T07:56:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2837177344 bytes 2024-04-06T07:56:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2881812480 bytes 2024-04-06T07:56:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2914948096 bytes 2024-04-06T07:56:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2950647808 bytes 2024-04-06T07:56:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 3002825728 bytes 2024-04-06T07:56:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 3015089152 bytes 2024-04-06T07:56:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 3021693952 bytes 2024-04-06T07:56:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 3098217472 bytes 2024-04-06T07:56:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 3116938240 bytes 2024-04-06T07:56:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 3124259840 bytes 2024-04-06T07:56:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 3174674432 bytes 2024-04-06T07:56:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 3218430976 bytes 2024-04-06T07:56:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 3238910976 bytes 2024-04-06T07:56:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7900 IOs, 3288968192 bytes 2024-04-06T07:56:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8000 IOs, 3320607744 bytes 2024-04-06T07:56:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8100 IOs, 3338865664 bytes 2024-04-06T07:56:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8200 IOs, 3363481600 bytes 2024-04-06T07:56:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8300 IOs, 3412911104 bytes 2024-04-06T07:56:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8400 IOs, 3470097408 bytes 2024-04-06T07:56:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8500 IOs, 3532997632 bytes 2024-04-06T07:56:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8600 IOs, 3545095168 bytes 2024-04-06T07:56:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8700 IOs, 3573236736 bytes 2024-04-06T07:56:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8800 IOs, 3617558528 bytes 2024-04-06T07:56:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T07:56:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:56:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T07:56:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T07:56:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:56:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:56:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:56:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:56:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:56:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:56:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T07:56:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":17830502.4,"bytes_per_snapshot":267457536.0,"last_snapshot_bytes":534915072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390194,"remote_snapshot_timestamp":1712390194,"replay_state":"idle"} 2024-04-06T07:56:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.0 on smithi129 2024-04-06T07:56:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:56:44 2024-04-06T07:56:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:56:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:56:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T07:56:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T07:56:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:56:45' 2024-04-06T07:56:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:56:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T07:56:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T07:56:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":17830502.4,"bytes_per_snapshot":267457536.0,"last_snapshot_bytes":534915072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390194,"remote_snapshot_timestamp":1712390194,"replay_state":"idle"} 2024-04-06T07:56:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:56:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:56:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T07:56:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:56:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T07:56:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.0 on smithi129 2024-04-06T07:56:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T07:56:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T07:56:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:56:44' 2024-04-06T07:56:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 07:56:44' 2024-04-06T07:56:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:56:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:56:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 07:56:44' 2024-04-06T07:56:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:56:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:56:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T07:56:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK42154262mirror.0smithi12918.2.2-1059-g4d05f4catrueOK42214263mirror.3smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":17830502.4,"bytes_per_snapshot":267457536.0,"last_snapshot_bytes":534915072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390194,"remote_snapshot_timestamp":1712390194,"replay_state":"idle"}42154262mirror.0smithi1292024-04-06 07:56:44cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 07:56:45' 2024-04-06T07:56:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T07:56:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T07:56:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:56:44' 2024-04-06T07:56:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T07:56:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T07:56:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:56:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T07:56:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:56:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:56:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap2 2024-04-06T07:56:57.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap2 2024-04-06T07:56:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:56:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:56:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:56:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2 2024-04-06T07:56:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-04-06T07:57:13.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9716968280460008) > weight (0.25) 2024-04-06T07:57:13.900 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6349677242564402) > weight (0.25) 2024-04-06T07:57:13.901 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2939151991940995) > weight (0.25) 2024-04-06T07:57:13.901 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.385063943251826) > weight (0.25) 2024-04-06T07:57:13.901 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-04-06T07:57:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T07:57:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:57:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:57:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T07:57:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T07:57:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T07:57:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T07:57:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:57:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T07:57:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:57:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:57:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:57:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:57:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:57:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:57:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:57:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:57:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:57:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T07:57:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T07:57:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T07:57:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T07:57:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T07:57:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T07:57:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T07:57:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T07:57:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:57:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:57:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T07:57:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:57:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T07:57:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T07:57:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:57:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T07:57:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T07:57:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 9 2024-04-06T07:57:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:57:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:57:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T07:57:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:57:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T07:57:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=9 2024-04-06T07:57:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:57:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:57:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T07:57:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:57:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:57:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:57:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:57:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:57:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:57:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:57:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:57:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 9 = '' 2024-04-06T07:57:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:57:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T07:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:57:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:57:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:57:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:57:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:57:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:57:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:57:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 9 = '' 2024-04-06T07:57:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:57:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T07:57:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:57:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:57:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:57:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:57:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:57:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:57:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:57:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:57:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 9 = '' 2024-04-06T07:57:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:57:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T07:57:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:57:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:57:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:57:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:57:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=9 2024-04-06T07:57:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 9 = 9 2024-04-06T07:57:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-04-06T07:57:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2 2024-04-06T07:57:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:57:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T07:57:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:57:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-04-06T07:57:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T07:57:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T07:57:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 4 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T07:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:57:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap1-clean ']' 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1-clean 2024-04-06T07:57:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T07:57:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:57:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:57:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:57:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:57:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap2 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:57:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:57:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:57:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:57:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap2-clean 2024-04-06T07:57:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-04-06T07:57:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2-clean 2024-04-06T07:57:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T07:57:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T07:57:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T07:57:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T07:57:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T07:57:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:57:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T07:57:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 26617856 bytes 2024-04-06T07:57:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 102986752 bytes 2024-04-06T07:57:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 161034240 bytes 2024-04-06T07:57:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 195947520 bytes 2024-04-06T07:57:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 238166016 bytes 2024-04-06T07:57:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 279349248 bytes 2024-04-06T07:57:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 321847296 bytes 2024-04-06T07:57:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 335468544 bytes 2024-04-06T07:57:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 382848000 bytes 2024-04-06T07:57:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 411515904 bytes 2024-04-06T07:57:38.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 472090624 bytes 2024-04-06T07:57:39.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 488582144 bytes 2024-04-06T07:57:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 555634688 bytes 2024-04-06T07:57:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 557020160 bytes 2024-04-06T07:57:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 557645824 bytes 2024-04-06T07:57:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 564978688 bytes 2024-04-06T07:57:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 617728000 bytes 2024-04-06T07:57:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 651278336 bytes 2024-04-06T07:57:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 756605952 bytes 2024-04-06T07:57:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 850665472 bytes 2024-04-06T07:57:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 867025920 bytes 2024-04-06T07:57:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 893257728 bytes 2024-04-06T07:57:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 936680448 bytes 2024-04-06T07:57:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 937290752 bytes 2024-04-06T07:57:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 971119616 bytes 2024-04-06T07:57:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1037619200 bytes 2024-04-06T07:57:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1138269184 bytes 2024-04-06T07:57:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1180828672 bytes 2024-04-06T07:57:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1222729728 bytes 2024-04-06T07:57:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1254054912 bytes 2024-04-06T07:57:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1271447552 bytes 2024-04-06T07:57:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1374662656 bytes 2024-04-06T07:57:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1432029184 bytes 2024-04-06T07:57:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1479056384 bytes 2024-04-06T07:57:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1522423808 bytes 2024-04-06T07:57:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1527862272 bytes 2024-04-06T07:57:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1585927168 bytes 2024-04-06T07:57:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1628958720 bytes 2024-04-06T07:57:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1658721280 bytes 2024-04-06T07:57:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1706330112 bytes 2024-04-06T07:57:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1758319616 bytes 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:57:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:57:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:57:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:57:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:57:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:57:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:57:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:57:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:57:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:57:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:57:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":17836373.33,"bytes_per_snapshot":535003136.0,"last_snapshot_bytes":535091200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390245,"remote_snapshot_timestamp":1712390245,"replay_state":"idle"} 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.0 on smithi129 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:57:44 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:57:45' 2024-04-06T07:57:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:57:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T07:57:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T07:57:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":17836373.33,"bytes_per_snapshot":535003136.0,"last_snapshot_bytes":535091200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390245,"remote_snapshot_timestamp":1712390245,"replay_state":"idle"} 2024-04-06T07:57:45.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:57:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:57:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T07:57:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:57:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T07:57:45.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.0 on smithi129 2024-04-06T07:57:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T07:57:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T07:57:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:57:44' 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 07:57:44' 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 07:57:44' 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:57:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T07:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK42154262mirror.0smithi12918.2.2-1059-g4d05f4catrueOK42214263mirror.3smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":17836373.33,"bytes_per_snapshot":535003136.0,"last_snapshot_bytes":535091200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390245,"remote_snapshot_timestamp":1712390245,"replay_state":"idle"}42154262mirror.0smithi1292024-04-06 07:57:44cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 07:57:45' 2024-04-06T07:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T07:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T07:57:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:57:44' 2024-04-06T07:57:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T07:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T07:57:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:57:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T07:57:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap3 2024-04-06T07:57:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap3 2024-04-06T07:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3 2024-04-06T07:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T07:58:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:58:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:58:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:58:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T07:58:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T07:58:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T07:58:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:58:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T07:58:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:58:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:58:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:58:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:58:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:58:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:58:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:58:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:58:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:58:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T07:58:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T07:58:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T07:58:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T07:58:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T07:58:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T07:58:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:58:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:58:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T07:58:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T07:58:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:58:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T07:58:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T07:58:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 13 2024-04-06T07:58:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:58:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:58:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:58:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T07:58:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T07:58:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=13 2024-04-06T07:58:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:58:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:58:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T07:58:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:58:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:58:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:58:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:58:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 13 = '' 2024-04-06T07:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T07:58:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:58:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:58:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:58:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:58:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:58:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:58:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:58:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 13 = '' 2024-04-06T07:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:58:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T07:58:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:58:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:58:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:58:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:58:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:58:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:58:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:58:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:58:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 13 = '' 2024-04-06T07:58:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:58:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T07:58:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:58:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:58:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:58:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:58:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=13 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 13 = 13 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T07:58:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:58:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-04-06T07:58:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 6 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 07:57:32 2024 2024-04-06T07:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T07:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T07:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T07:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T07:58:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap2-clean ']' 2024-04-06T07:58:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2-clean 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:58:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:58:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:58:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap3 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:58:23.901 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T07:58:23.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T07:58:23.902 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T07:58:23.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T07:58:23.896+0000 7f1b31052640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 28944) UID: 1000 2024-04-06T07:58:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:58:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:58:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:58:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap3-clean 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3-clean 2024-04-06T07:58:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T07:58:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T07:58:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T07:58:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T07:58:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=10 2024-04-06T07:58:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T07:58:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T07:58:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 623616 bytes 2024-04-06T07:58:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 33356800 bytes 2024-04-06T07:58:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 53965824 bytes 2024-04-06T07:58:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 76626944 bytes 2024-04-06T07:58:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 82512896 bytes 2024-04-06T07:58:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 95954944 bytes 2024-04-06T07:58:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 105370624 bytes 2024-04-06T07:58:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 145416192 bytes 2024-04-06T07:58:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 179233792 bytes 2024-04-06T07:58:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 240631808 bytes 2024-04-06T07:58:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 309936128 bytes 2024-04-06T07:58:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 341122048 bytes 2024-04-06T07:58:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 365478912 bytes 2024-04-06T07:58:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 405094400 bytes 2024-04-06T07:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 448670720 bytes 2024-04-06T07:58:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 513987584 bytes 2024-04-06T07:58:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 578401280 bytes 2024-04-06T07:58:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 677923840 bytes 2024-04-06T07:58:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 741253120 bytes 2024-04-06T07:58:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 796309504 bytes 2024-04-06T07:58:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 842162176 bytes 2024-04-06T07:58:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 893334528 bytes 2024-04-06T07:58:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 909392896 bytes 2024-04-06T07:58:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 951435264 bytes 2024-04-06T07:58:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 997516288 bytes 2024-04-06T07:58:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1082969088 bytes 2024-04-06T07:58:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1158993920 bytes 2024-04-06T07:58:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1199335424 bytes 2024-04-06T07:58:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1223537664 bytes 2024-04-06T07:58:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1277931520 bytes 2024-04-06T07:58:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1363298304 bytes 2024-04-06T07:58:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1419489280 bytes 2024-04-06T07:58:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1457923072 bytes 2024-04-06T07:58:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1458503680 bytes 2024-04-06T07:58:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1502903296 bytes 2024-04-06T07:58:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1585054720 bytes 2024-04-06T07:58:32.902 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T07:58:32.903 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T07:58:32.903 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T07:58:32.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T07:58:32.903 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T07:58:32.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T07:58:32.907 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-04-06T07:58:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1627823104 bytes 2024-04-06T07:58:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1702173696 bytes 2024-04-06T07:58:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1724395520 bytes 2024-04-06T07:58:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1749139456 bytes 2024-04-06T07:58:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1773057024 bytes 2024-04-06T07:58:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1798363136 bytes 2024-04-06T07:58:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1883572224 bytes 2024-04-06T07:58:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1941379072 bytes 2024-04-06T07:58:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1941996544 bytes 2024-04-06T07:58:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2024659968 bytes 2024-04-06T07:58:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2068320256 bytes 2024-04-06T07:58:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2097214464 bytes 2024-04-06T07:58:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2110638080 bytes 2024-04-06T07:58:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2131397632 bytes 2024-04-06T07:58:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2132013056 bytes 2024-04-06T07:58:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2146514944 bytes 2024-04-06T07:58:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2192745472 bytes 2024-04-06T07:58:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2210616320 bytes 2024-04-06T07:58:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2285726720 bytes 2024-04-06T07:58:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2306854912 bytes 2024-04-06T07:58:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2330631168 bytes 2024-04-06T07:58:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2395768832 bytes 2024-04-06T07:58:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2475405312 bytes 2024-04-06T07:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2498524160 bytes 2024-04-06T07:58:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2549293056 bytes 2024-04-06T07:58:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T07:58:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T07:58:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T07:58:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T07:58:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:58:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:58:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:58:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:58:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T07:58:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T07:58:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:58:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T07:58:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535003136.0,"last_snapshot_bytes":535091200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390245,"remote_snapshot_timestamp":1712390245,"replay_state":"idle"} 2024-04-06T07:58:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.0 on smithi129 2024-04-06T07:58:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 07:58:14 2024-04-06T07:58:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T07:58:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T07:58:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: down+unknown 2024-04-06T07:58:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: status not found 2024-04-06T07:58:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: ' 2024-04-06T07:58:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T07:58:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T07:58:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T07:58:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535003136.0,"last_snapshot_bytes":535091200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390245,"remote_snapshot_timestamp":1712390245,"replay_state":"idle"} 2024-04-06T07:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T07:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:58:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T07:58:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:58:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T07:58:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.0 on smithi129 2024-04-06T07:58:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T07:58:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:58:14' 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 07:58:14' 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 07:58:14' 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T07:58:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T07:58:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T07:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='WARNINGOKWARNING142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK42154262mirror.0smithi12918.2.2-1059-g4d05f4catrueOK42214263mirror.3smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535003136.0,"last_snapshot_bytes":535091200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390245,"remote_snapshot_timestamp":1712390245,"replay_state":"idle"}42154262mirror.0smithi1292024-04-06 07:58:14cluster23aea22be-a2a5-4341-813f-81096758e7a9down+unknownstatus not found' 2024-04-06T07:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T07:58:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T07:58:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 07:58:14' 2024-04-06T07:58:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T07:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T07:58:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T07:58:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T07:58:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T07:58:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap4 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap4 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4 2024-04-06T07:58:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-04-06T07:59:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T07:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:59:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:59:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T07:59:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T07:59:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T07:59:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T07:59:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T07:59:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T07:59:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T07:59:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T07:59:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T07:59:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T07:59:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T07:59:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T07:59:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T07:59:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T07:59:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T07:59:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:59:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:59:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T07:59:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T07:59:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T07:59:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T07:59:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T07:59:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T07:59:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T07:59:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T07:59:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T07:59:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:59:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T07:59:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:59:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:59:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T07:59:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:59:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T07:59:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T07:59:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T07:59:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T07:59:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:59:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T07:59:08.902 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T07:59:08.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T07:59:08.903 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T07:59:08.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T07:59:08.895+0000 7f3735ff4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 28935) UID: 1000 2024-04-06T07:59:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 17 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T07:59:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T07:59:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=17 2024-04-06T07:59:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T07:59:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T07:59:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T07:59:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T07:59:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 2 2024-04-06T07:59:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 2 2024-04-06T07:59:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 4 2024-04-06T07:59:17.903 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T07:59:17.904 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T07:59:17.904 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T07:59:17.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T07:59:17.904 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T07:59:17.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T07:59:17.907 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-04-06T07:59:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 4 2024-04-06T07:59:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 8 2024-04-06T07:59:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 8 2024-04-06T07:59:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T07:59:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T07:59:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T07:59:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T07:59:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T07:59:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = '' 2024-04-06T07:59:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T07:59:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 16 2024-04-06T07:59:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T07:59:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T07:59:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T07:59:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T07:59:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=17 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 17 = 17 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4 2024-04-06T07:59:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T07:59:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T07:59:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T07:59:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-04-06T07:59:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T07:59:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T07:59:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T07:59:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 8 2024-04-06T07:59:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T07:59:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 07:57:32 2024 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T07:59:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T07:59:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T07:59:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap3-clean ']' 2024-04-06T07:59:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-06T07:59:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T07:59:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T07:59:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3-clean 2024-04-06T07:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T07:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:59:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T07:59:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:59:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T07:59:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T07:59:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:00:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap4 2024-04-06T08:00:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4 2024-04-06T08:00:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:00:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:00:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:00:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:00:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:02.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:00:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap4-clean 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4-clean 2024-04-06T08:00:02.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:00:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:00:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T08:00:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:00:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:00:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:00:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:00:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T08:00:03.017 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:00:03.011+0000 7f5a15134640 -1 reset not still connected to 0x55db89d0e4e0 2024-04-06T08:00:03.020 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:00:03.011+0000 7f5a15134640 -1 reset not still connected to 0x55db8ebbd1e0 2024-04-06T08:00:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 27133952 bytes 2024-04-06T08:00:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 97436672 bytes 2024-04-06T08:00:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 103603200 bytes 2024-04-06T08:00:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 155717632 bytes 2024-04-06T08:00:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 191135744 bytes 2024-04-06T08:00:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 223025152 bytes 2024-04-06T08:00:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 296994816 bytes 2024-04-06T08:00:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 351553536 bytes 2024-04-06T08:00:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 395124736 bytes 2024-04-06T08:00:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 395764736 bytes 2024-04-06T08:00:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 435614720 bytes 2024-04-06T08:00:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 471278592 bytes 2024-04-06T08:00:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 492690432 bytes 2024-04-06T08:00:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 518866944 bytes 2024-04-06T08:00:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 539570176 bytes 2024-04-06T08:00:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 595005440 bytes 2024-04-06T08:00:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 724783104 bytes 2024-04-06T08:00:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 756178944 bytes 2024-04-06T08:00:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 774057984 bytes 2024-04-06T08:00:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 809183232 bytes 2024-04-06T08:00:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 860288000 bytes 2024-04-06T08:00:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 899418112 bytes 2024-04-06T08:00:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 926633984 bytes 2024-04-06T08:00:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 989070336 bytes 2024-04-06T08:00:07.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1048174592 bytes 2024-04-06T08:00:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1082003456 bytes 2024-04-06T08:00:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1103001600 bytes 2024-04-06T08:00:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1141865472 bytes 2024-04-06T08:00:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1226901504 bytes 2024-04-06T08:00:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1242056704 bytes 2024-04-06T08:00:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1272663040 bytes 2024-04-06T08:00:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1317609472 bytes 2024-04-06T08:00:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1414957056 bytes 2024-04-06T08:00:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1464603648 bytes 2024-04-06T08:00:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1477756928 bytes 2024-04-06T08:00:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1523973120 bytes 2024-04-06T08:00:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1540877312 bytes 2024-04-06T08:00:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1590889472 bytes 2024-04-06T08:00:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1643699200 bytes 2024-04-06T08:00:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1709400064 bytes 2024-04-06T08:00:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1782005760 bytes 2024-04-06T08:00:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1782640640 bytes 2024-04-06T08:00:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1801011200 bytes 2024-04-06T08:00:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1866369024 bytes 2024-04-06T08:00:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1891275776 bytes 2024-04-06T08:00:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1931675648 bytes 2024-04-06T08:00:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1993647104 bytes 2024-04-06T08:00:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2055579648 bytes 2024-04-06T08:00:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2079510528 bytes 2024-04-06T08:00:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2083700736 bytes 2024-04-06T08:00:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2141612032 bytes 2024-04-06T08:00:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2174173184 bytes 2024-04-06T08:00:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2206044160 bytes 2024-04-06T08:00:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2261740544 bytes 2024-04-06T08:00:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2344562688 bytes 2024-04-06T08:00:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2350636032 bytes 2024-04-06T08:00:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2377761792 bytes 2024-04-06T08:00:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2486494208 bytes 2024-04-06T08:00:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2507014144 bytes 2024-04-06T08:00:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2507616256 bytes 2024-04-06T08:00:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2528880640 bytes 2024-04-06T08:00:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2612078592 bytes 2024-04-06T08:00:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2623696896 bytes 2024-04-06T08:00:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2648916992 bytes 2024-04-06T08:00:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2711499776 bytes 2024-04-06T08:00:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2718066688 bytes 2024-04-06T08:00:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2791174144 bytes 2024-04-06T08:00:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2826762240 bytes 2024-04-06T08:00:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2871301120 bytes 2024-04-06T08:00:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2959208448 bytes 2024-04-06T08:00:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2959804416 bytes 2024-04-06T08:00:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2990879744 bytes 2024-04-06T08:00:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:00:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:00:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:00:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:00:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T08:00:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:00:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":17876548.27,"bytes_per_snapshot":536296448.0,"last_snapshot_bytes":536296448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390347,"remote_snapshot_timestamp":1712390347,"replay_state":"idle"} 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.3 on smithi129 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:00:15 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T08:00:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:00:14' 2024-04-06T08:00:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T08:00:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T08:00:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T08:00:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":17876548.27,"bytes_per_snapshot":536296448.0,"last_snapshot_bytes":536296448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390347,"remote_snapshot_timestamp":1712390347,"replay_state":"idle"} 2024-04-06T08:00:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:00:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:00:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T08:00:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:00:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T08:00:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.3 on smithi129 2024-04-06T08:00:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T08:00:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T08:00:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:00:15' 2024-04-06T08:00:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 08:00:15' 2024-04-06T08:00:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:00:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:00:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 08:00:15' 2024-04-06T08:00:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:00:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:00:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T08:00:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK42214263mirror.3smithi12918.2.2-1059-g4d05f4catrueOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOK44284431mirror.0smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":17876548.27,"bytes_per_snapshot":536296448.0,"last_snapshot_bytes":536296448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390347,"remote_snapshot_timestamp":1712390347,"replay_state":"idle"}42214263mirror.3smithi1292024-04-06 08:00:15cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 08:00:14' 2024-04-06T08:00:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T08:00:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T08:00:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:00:15' 2024-04-06T08:00:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T08:00:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T08:00:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:00:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T08:00:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap5 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap5 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5 2024-04-06T08:00:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:00:29.908 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T08:00:29.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T08:00:29.908 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T08:00:29.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:00:29.906+0000 7fba88156640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 31642) UID: 1000 2024-04-06T08:00:34.909 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T08:00:34.910 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:00:34.910 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T08:00:34.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T08:00:34.911 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T08:00:34.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T08:00:34.915 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-04-06T08:00:37.908 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3633279700947366) > weight (0.25) 2024-04-06T08:00:37.908 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6632875474712913) > weight (0.25) 2024-04-06T08:00:37.908 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6025749833174339) > weight (0.25) 2024-04-06T08:00:37.909 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T08:00:37.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T08:00:37.909 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T08:00:37.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:00:37.906+0000 7efe85806640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 28941) UID: 1000 2024-04-06T08:00:42.910 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T08:00:42.911 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:00:42.911 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T08:00:42.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T08:00:42.911 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T08:00:42.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T08:00:42.915 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-04-06T08:00:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:00:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:00:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:00:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:00:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:00:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:00:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:00:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:00:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:00:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:00:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:00:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:00:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:00:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:00:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:00:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:00:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:00:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T08:00:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:00:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:00:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:00:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:00:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:00:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:00:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:00:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:00:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:00:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T08:00:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:00:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:00:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:00:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:00:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:00:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T08:00:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 1 2024-04-06T08:00:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:00:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:00:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:00:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:00:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:00:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:00:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:00:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:00:47.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:00:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:00:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:00:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:00:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:00:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:00:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:00:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:00:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:00:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T08:00:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T08:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:00:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:00:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:00:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:00:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T08:00:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 2 2024-04-06T08:00:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:00:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:00:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:00:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:00:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:00:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:00:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:00:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:00:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:00:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:00:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:00:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:00:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:00:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:00:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:00:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:00:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T08:00:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:00:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:00:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:00:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:00:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T08:00:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 4 2024-04-06T08:00:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:00:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:00:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:00:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:00:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:00:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:00:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:00:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:00:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:00:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:00:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:00:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:00:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:00:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:00:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:00:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:00:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:00:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:00:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:00:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T08:00:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:00:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:00:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:00:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:00:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:00:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T08:00:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:00:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:00:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:00:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:00:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 8 2024-04-06T08:01:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:01:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:01:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:01:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:01:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:01:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:01:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:01:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:01:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:01:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:01:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:01:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:01:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:01:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:01:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:01:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:01:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:01:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:01:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:01:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:01:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:01:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:01:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:01:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:01:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:01:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:01:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:01:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:01:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:01:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:01:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:01:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:01:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:01:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:01:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:01:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:01:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:01:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:01:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:01:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:01:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:01:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:01:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:01:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:01:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:01:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:01:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-06T08:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:01:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:01:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:01:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:01:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:01:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:01:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:01:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:01:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:01:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:01:03.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-06T08:01:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:01:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:01:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:01:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:01:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 8 2024-04-06T08:01:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:01:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:01:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:01:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:01:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:01:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:01:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:01:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:01:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:01:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:01:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:01:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:01:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:01:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:01:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:01:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:01:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:01:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:01:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:01:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:01:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:01:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:01:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:01:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:01:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:01:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:01:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:01:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:01:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:01:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:01:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:01:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:01:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:01:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:01:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:01:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:01:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:01:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:01:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:01:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:01:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:01:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 21 2024-04-06T08:01:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:01:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:01:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:01:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:01:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:01:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=21 2024-04-06T08:01:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:01:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:01:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:01:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:01:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:01:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:01:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:01:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:01:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:01:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:01:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:01:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 21 = '' 2024-04-06T08:01:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:01:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:01:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:01:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:01:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 21 = '' 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:01:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:01:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:01:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:01:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:01:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 21 = '' 2024-04-06T08:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:01:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:01:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:01:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:01:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=21 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 21 = 21 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:01:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:01:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-04-06T08:01:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:01:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 10 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 07:59:58 2024 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:01:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:01:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:01:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap4-clean ']' 2024-04-06T08:01:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-06T08:01:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4-clean 2024-04-06T08:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:01:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:01:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:01:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap5 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:01:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:01:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:01:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap5-clean 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5-clean 2024-04-06T08:01:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:01:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:01:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T08:01:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:01:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:01:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:01:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T08:01:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 51237888 bytes 2024-04-06T08:01:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 90584064 bytes 2024-04-06T08:01:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 147638272 bytes 2024-04-06T08:01:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 205945856 bytes 2024-04-06T08:01:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 279932928 bytes 2024-04-06T08:01:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 348567552 bytes 2024-04-06T08:01:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 356162560 bytes 2024-04-06T08:01:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 403041280 bytes 2024-04-06T08:01:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 477590528 bytes 2024-04-06T08:01:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 496733184 bytes 2024-04-06T08:01:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 518164480 bytes 2024-04-06T08:01:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 561209344 bytes 2024-04-06T08:01:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 574279680 bytes 2024-04-06T08:01:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 595939328 bytes 2024-04-06T08:01:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 631028736 bytes 2024-04-06T08:01:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 687826944 bytes 2024-04-06T08:01:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 756625408 bytes 2024-04-06T08:01:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 802692096 bytes 2024-04-06T08:01:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 829116416 bytes 2024-04-06T08:01:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 897612800 bytes 2024-04-06T08:01:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 906863616 bytes 2024-04-06T08:01:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 947780608 bytes 2024-04-06T08:01:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1032745984 bytes 2024-04-06T08:01:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1071171584 bytes 2024-04-06T08:01:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1106441216 bytes 2024-04-06T08:01:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1134086144 bytes 2024-04-06T08:01:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1148482560 bytes 2024-04-06T08:01:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1206371328 bytes 2024-04-06T08:01:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1215976448 bytes 2024-04-06T08:01:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1268810752 bytes 2024-04-06T08:01:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1319598080 bytes 2024-04-06T08:01:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1341069312 bytes 2024-04-06T08:01:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1383809024 bytes 2024-04-06T08:01:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1479441408 bytes 2024-04-06T08:01:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1525373952 bytes 2024-04-06T08:01:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1575984128 bytes 2024-04-06T08:01:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1691297792 bytes 2024-04-06T08:01:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1691938816 bytes 2024-04-06T08:01:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1746170880 bytes 2024-04-06T08:01:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1841427456 bytes 2024-04-06T08:01:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1908310016 bytes 2024-04-06T08:01:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1953932288 bytes 2024-04-06T08:01:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2010218496 bytes 2024-04-06T08:01:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2063101952 bytes 2024-04-06T08:01:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2131099648 bytes 2024-04-06T08:01:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2154446848 bytes 2024-04-06T08:01:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2213646336 bytes 2024-04-06T08:01:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2256321536 bytes 2024-04-06T08:01:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2315035648 bytes 2024-04-06T08:01:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2342683648 bytes 2024-04-06T08:01:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2416322560 bytes 2024-04-06T08:01:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2431611904 bytes 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:01:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:01:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:01:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:01:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:01:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:01:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:01:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T08:01:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390347,"remote_snapshot_timestamp":1712390347,"replay_state":"idle"} 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.1 on smithi129 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:01:14 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:01:14' 2024-04-06T08:01:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T08:01:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T08:01:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T08:01:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390347,"remote_snapshot_timestamp":1712390347,"replay_state":"idle"} 2024-04-06T08:01:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:01:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:01:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T08:01:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:01:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T08:01:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.1 on smithi129 2024-04-06T08:01:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T08:01:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:01:14' 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 08:01:14' 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 08:01:14' 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:01:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:01:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T08:01:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOK44284431mirror.0smithi12918.2.2-1059-g4d05f4catrueOK44734476mirror.3smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390347,"remote_snapshot_timestamp":1712390347,"replay_state":"idle"}42144269mirror.1smithi1292024-04-06 08:01:14cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 08:01:14' 2024-04-06T08:01:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T08:01:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T08:01:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:01:14' 2024-04-06T08:01:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T08:01:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T08:01:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:01:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T08:01:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap6 2024-04-06T08:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap6 2024-04-06T08:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:01:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:01:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:01:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6 2024-04-06T08:01:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:01:55.917 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7092909909873102) > weight (0.25) 2024-04-06T08:01:55.917 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4836807763676172) > weight (0.25) 2024-04-06T08:01:55.917 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T08:01:55.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T08:01:55.918 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T08:01:55.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:01:55.918+0000 7f35ac6d2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 29314) UID: 1000 2024-04-06T08:01:58.919 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T08:01:58.919 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:01:58.920 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T08:01:58.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T08:01:58.920 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T08:01:58.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T08:01:58.924 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:02:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:02:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:02:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:02:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:02:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:02:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:02:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:02:03.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:02:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:02:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:02:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:02:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:02:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:02:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:02:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:02:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:02:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:02:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:02:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:02:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:02:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:02:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:02:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:02:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:02:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:02:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:02:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:02:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:02:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:02:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:02:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 25 2024-04-06T08:02:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:02:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:02:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:02:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:02:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=25 2024-04-06T08:02:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:02:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:02:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:02:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:02:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 25 = '' 2024-04-06T08:02:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:02:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:02:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:02:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:02:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 25 = '' 2024-04-06T08:02:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:02:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 25 = '' 2024-04-06T08:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:02:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=25 2024-04-06T08:02:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 25 = 25 2024-04-06T08:02:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-04-06T08:02:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6 2024-04-06T08:02:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:02:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:02:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:02:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-04-06T08:02:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:02:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:02:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:02:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 12 2024-04-06T08:02:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:01:18 2024 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:02:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:02:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap5-clean ']' 2024-04-06T08:02:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5-clean 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:02:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:02:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:02:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap6 2024-04-06T08:02:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6 2024-04-06T08:02:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:02:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:02:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:02:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:02:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:02:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap6-clean 2024-04-06T08:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-04-06T08:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6-clean 2024-04-06T08:02:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:02:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:02:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T08:02:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:02:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=5 2024-04-06T08:02:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:02:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T08:02:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 15162368 bytes 2024-04-06T08:02:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 42359808 bytes 2024-04-06T08:02:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 111635456 bytes 2024-04-06T08:02:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 134893568 bytes 2024-04-06T08:02:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 204943360 bytes 2024-04-06T08:02:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 289096704 bytes 2024-04-06T08:02:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 322990080 bytes 2024-04-06T08:02:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 387511296 bytes 2024-04-06T08:02:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 425717760 bytes 2024-04-06T08:02:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 489744384 bytes 2024-04-06T08:02:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 544489472 bytes 2024-04-06T08:02:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 610240512 bytes 2024-04-06T08:02:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 658555904 bytes 2024-04-06T08:02:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 692959232 bytes 2024-04-06T08:02:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 724514816 bytes 2024-04-06T08:02:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 821186560 bytes 2024-04-06T08:02:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 846120960 bytes 2024-04-06T08:02:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 927709184 bytes 2024-04-06T08:02:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 992531456 bytes 2024-04-06T08:02:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 1096102912 bytes 2024-04-06T08:02:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1168100352 bytes 2024-04-06T08:02:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1254126592 bytes 2024-04-06T08:02:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1350754304 bytes 2024-04-06T08:02:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1383368704 bytes 2024-04-06T08:02:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1403198464 bytes 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:02:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:02:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:02:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:02:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:02:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:02:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:02:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":17895150.93,"bytes_per_snapshot":536357888.0,"last_snapshot_bytes":536854528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390524,"remote_snapshot_timestamp":1712390524,"replay_state":"idle"} 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.1 on smithi129 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:02:14 2024-04-06T08:02:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T08:02:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T08:02:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T08:02:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T08:02:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:02:14' 2024-04-06T08:02:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T08:02:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T08:02:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T08:02:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":17895150.93,"bytes_per_snapshot":536357888.0,"last_snapshot_bytes":536854528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390524,"remote_snapshot_timestamp":1712390524,"replay_state":"idle"} 2024-04-06T08:02:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:02:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:02:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T08:02:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:02:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T08:02:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.1 on smithi129 2024-04-06T08:02:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T08:02:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T08:02:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:02:14' 2024-04-06T08:02:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 08:02:14' 2024-04-06T08:02:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 08:02:14' 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:02:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T08:02:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOK44284431mirror.0smithi12918.2.2-1059-g4d05f4catrueOK44734476mirror.3smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":17895150.93,"bytes_per_snapshot":536357888.0,"last_snapshot_bytes":536854528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390524,"remote_snapshot_timestamp":1712390524,"replay_state":"idle"}42144269mirror.1smithi1292024-04-06 08:02:14cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 08:02:14' 2024-04-06T08:02:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T08:02:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T08:02:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:02:14' 2024-04-06T08:02:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T08:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T08:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:02:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T08:02:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap7 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap7 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7 2024-04-06T08:02:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:02:29.915 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9306920073432033) > weight (0.25) 2024-04-06T08:02:29.916 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9622090705415915) > weight (0.25) 2024-04-06T08:02:29.916 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7730672666932151) > weight (0.25) 2024-04-06T08:02:29.916 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9694486642389804) > weight (0.25) 2024-04-06T08:02:29.916 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-06T08:02:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:02:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:02:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:02:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:02:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:02:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:02:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:02:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:02:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:02:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:02:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:02:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:02:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:02:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:02:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:02:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:02:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:02:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:02:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:02:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:02:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:02:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:02:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:02:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:02:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:02:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:02:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:02:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:02:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:02:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:02:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:02:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:02:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:02:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:02:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:02:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:02:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:02:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:02:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:02:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:02:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:02:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:02:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:02:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:02:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:02:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:02:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:02:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:02:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:02:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:02:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 29 2024-04-06T08:02:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:02:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:02:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:02:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:02:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:02:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=29 2024-04-06T08:02:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:02:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:02:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:02:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:02:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:02:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 29 = '' 2024-04-06T08:02:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:02:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:02:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:02:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:02:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:02:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:02:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 29 = '' 2024-04-06T08:02:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:02:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:02:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:02:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:02:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:02:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:02:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=29 2024-04-06T08:02:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 29 = 29 2024-04-06T08:02:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:02:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:02:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-04-06T08:02:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:02:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:02:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:02:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 14 2024-04-06T08:02:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:02:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:01:18 2024 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:02:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:02:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:02:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap6-clean ']' 2024-04-06T08:02:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-06T08:02:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6-clean 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:02:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:02:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:02:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:02:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap7 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:03:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:03:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:03:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap7-clean 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7-clean 2024-04-06T08:03:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:03:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:03:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T08:03:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:03:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:03:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:03:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T08:03:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 66045952 bytes 2024-04-06T08:03:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 72340480 bytes 2024-04-06T08:03:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 88488960 bytes 2024-04-06T08:03:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 151786496 bytes 2024-04-06T08:03:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 203284480 bytes 2024-04-06T08:03:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 249670656 bytes 2024-04-06T08:03:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 293137408 bytes 2024-04-06T08:03:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 407582720 bytes 2024-04-06T08:03:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 461721600 bytes 2024-04-06T08:03:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 502691840 bytes 2024-04-06T08:03:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 532673536 bytes 2024-04-06T08:03:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 580789248 bytes 2024-04-06T08:03:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 581419008 bytes 2024-04-06T08:03:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 644174848 bytes 2024-04-06T08:03:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 661767168 bytes 2024-04-06T08:03:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 701123584 bytes 2024-04-06T08:03:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 769218560 bytes 2024-04-06T08:03:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 790666240 bytes 2024-04-06T08:03:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 792204288 bytes 2024-04-06T08:03:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 812241920 bytes 2024-04-06T08:03:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 812868608 bytes 2024-04-06T08:03:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 904796160 bytes 2024-04-06T08:03:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 919522304 bytes 2024-04-06T08:03:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 989443072 bytes 2024-04-06T08:03:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1026260992 bytes 2024-04-06T08:03:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1069296640 bytes 2024-04-06T08:03:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1075377152 bytes 2024-04-06T08:03:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1101641728 bytes 2024-04-06T08:03:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1163480064 bytes 2024-04-06T08:03:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1229109248 bytes 2024-04-06T08:03:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1280838656 bytes 2024-04-06T08:03:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1297143808 bytes 2024-04-06T08:03:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1348024320 bytes 2024-04-06T08:03:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1427919872 bytes 2024-04-06T08:03:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1474154496 bytes 2024-04-06T08:03:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1491298304 bytes 2024-04-06T08:03:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1499259904 bytes 2024-04-06T08:03:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1594048512 bytes 2024-04-06T08:03:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1635795968 bytes 2024-04-06T08:03:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1675125760 bytes 2024-04-06T08:03:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1759474688 bytes 2024-04-06T08:03:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1788267520 bytes 2024-04-06T08:03:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1841662976 bytes 2024-04-06T08:03:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1862938624 bytes 2024-04-06T08:03:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1922626560 bytes 2024-04-06T08:03:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1971079168 bytes 2024-04-06T08:03:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1977359360 bytes 2024-04-06T08:03:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2037442560 bytes 2024-04-06T08:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2080640000 bytes 2024-04-06T08:03:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2141253632 bytes 2024-04-06T08:03:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2206833664 bytes 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:03:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:03:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:03:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:03:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:03:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:03:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T08:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T08:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T08:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536357888.0,"last_snapshot_bytes":536854528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390524,"remote_snapshot_timestamp":1712390524,"replay_state":"idle"} 2024-04-06T08:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.1 on smithi129 2024-04-06T08:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:02:44 2024-04-06T08:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T08:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T08:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T08:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T08:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:02:44' 2024-04-06T08:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T08:03:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T08:03:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T08:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536357888.0,"last_snapshot_bytes":536854528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390524,"remote_snapshot_timestamp":1712390524,"replay_state":"idle"} 2024-04-06T08:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:03:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T08:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:03:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T08:03:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.1 on smithi129 2024-04-06T08:03:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T08:03:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:02:44' 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 08:02:44' 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 08:02:44' 2024-04-06T08:03:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:03:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:03:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T08:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOK44284431mirror.0smithi12918.2.2-1059-g4d05f4catrueOK44734476mirror.3smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536357888.0,"last_snapshot_bytes":536854528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390524,"remote_snapshot_timestamp":1712390524,"replay_state":"idle"}42144269mirror.1smithi1292024-04-06 08:02:44cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 08:02:44' 2024-04-06T08:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T08:03:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T08:03:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:02:44' 2024-04-06T08:03:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T08:03:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T08:03:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:03:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T08:03:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap8 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap8 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8 2024-04-06T08:03:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:03:19.925 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4502316305607911) > weight (0.25) 2024-04-06T08:03:19.925 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7560359685697725) > weight (0.25) 2024-04-06T08:03:19.925 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9379336746903661) > weight (0.25) 2024-04-06T08:03:19.925 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.47394146457097575) > weight (0.25) 2024-04-06T08:03:19.925 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-04-06T08:03:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:03:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:03:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:03:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:03:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:03:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:03:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:03:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:03:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:03:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:03:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:03:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:03:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:03:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:03:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:03:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:03:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:03:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:03:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:03:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:03:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:03:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:03:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:03:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:03:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:03:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:03:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:03:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:03:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:03:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:03:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:03:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:03:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:03:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:03:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:03:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:03:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:03:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:03:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:03:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:03:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:03:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:03:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:03:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:03:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:03:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:03:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:03:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:03:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:03:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:03:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:03:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:03:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:03:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:03:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:03:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:03:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:03:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:03:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 33 2024-04-06T08:03:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:03:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:03:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:03:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:03:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:03:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=33 2024-04-06T08:03:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:03:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:03:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:03:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:03:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:03:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:03:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:03:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:03:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:03:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:03:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:03:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 33 = '' 2024-04-06T08:03:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:03:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:03:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:03:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:03:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:03:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:03:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:03:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:03:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:03:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:03:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 33 = '' 2024-04-06T08:03:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:03:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:03:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:03:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:03:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:03:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:03:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:03:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:03:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:03:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:03:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 33 = '' 2024-04-06T08:03:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:03:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:03:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:03:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:03:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:03:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=33 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 33 = 33 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8 2024-04-06T08:03:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:03:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:03:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:03:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-04-06T08:03:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:03:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 16 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:02:58 2024 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:03:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:03:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:03:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:03:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:03:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap7-clean ']' 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7-clean 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:03:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:03:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:03:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:03:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:03:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap8 2024-04-06T08:03:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8 2024-04-06T08:03:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:03:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:03:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:03:53.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:03:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:03:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap8-clean 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8-clean 2024-04-06T08:03:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:03:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:03:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T08:03:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:03:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:03:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:03:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:03:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:03:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:03:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T08:03:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 73111552 bytes 2024-04-06T08:03:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 121938944 bytes 2024-04-06T08:03:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 170027008 bytes 2024-04-06T08:03:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 262947840 bytes 2024-04-06T08:03:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 281131008 bytes 2024-04-06T08:03:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 344832000 bytes 2024-04-06T08:03:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 398379008 bytes 2024-04-06T08:03:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 456632320 bytes 2024-04-06T08:03:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 527426560 bytes 2024-04-06T08:03:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 608829440 bytes 2024-04-06T08:03:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 685665280 bytes 2024-04-06T08:03:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 704432128 bytes 2024-04-06T08:03:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 723858432 bytes 2024-04-06T08:03:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 768273408 bytes 2024-04-06T08:03:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 789445632 bytes 2024-04-06T08:03:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 867485696 bytes 2024-04-06T08:03:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 933742592 bytes 2024-04-06T08:03:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 960400384 bytes 2024-04-06T08:03:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 1003031552 bytes 2024-04-06T08:03:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 1047369728 bytes 2024-04-06T08:03:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1069895680 bytes 2024-04-06T08:03:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1117437952 bytes 2024-04-06T08:03:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1179245568 bytes 2024-04-06T08:03:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1217632256 bytes 2024-04-06T08:03:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1279264768 bytes 2024-04-06T08:03:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1289185280 bytes 2024-04-06T08:03:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1318667264 bytes 2024-04-06T08:04:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1430743040 bytes 2024-04-06T08:04:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1488347136 bytes 2024-04-06T08:04:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1555737600 bytes 2024-04-06T08:04:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1623830528 bytes 2024-04-06T08:04:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1677072384 bytes 2024-04-06T08:04:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1750736896 bytes 2024-04-06T08:04:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1785047040 bytes 2024-04-06T08:04:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1799272448 bytes 2024-04-06T08:04:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1844332544 bytes 2024-04-06T08:04:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1924388864 bytes 2024-04-06T08:04:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 2000587776 bytes 2024-04-06T08:04:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 2039828480 bytes 2024-04-06T08:04:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 2126734336 bytes 2024-04-06T08:04:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 2154816512 bytes 2024-04-06T08:04:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 2194318336 bytes 2024-04-06T08:04:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2248615936 bytes 2024-04-06T08:04:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2249226240 bytes 2024-04-06T08:04:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2269008896 bytes 2024-04-06T08:04:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2314842112 bytes 2024-04-06T08:04:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2327127040 bytes 2024-04-06T08:04:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2365015040 bytes 2024-04-06T08:04:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2415248384 bytes 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:04:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:04:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:04:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:04:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:04:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T08:04:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:04:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:04:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:04:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:04:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:04:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514475008.0,"last_snapshot_bytes":492095488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390573,"remote_snapshot_timestamp":1712390573,"replay_state":"idle"} 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.1 on smithi129 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:03:44 2024-04-06T08:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T08:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T08:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T08:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T08:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:03:45' 2024-04-06T08:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T08:04:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T08:04:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T08:04:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514475008.0,"last_snapshot_bytes":492095488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390573,"remote_snapshot_timestamp":1712390573,"replay_state":"idle"} 2024-04-06T08:04:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:04:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:04:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T08:04:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:04:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T08:04:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.1 on smithi129 2024-04-06T08:04:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T08:04:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T08:04:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:03:44' 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 08:03:44' 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 08:03:44' 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:04:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T08:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4cafalseOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOK44284431mirror.0smithi12918.2.2-1059-g4d05f4catrueOK44734476mirror.3smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514475008.0,"last_snapshot_bytes":492095488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390573,"remote_snapshot_timestamp":1712390573,"replay_state":"idle"}42144269mirror.1smithi1292024-04-06 08:03:44cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 08:03:45' 2024-04-06T08:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T08:04:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T08:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:03:44' 2024-04-06T08:04:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T08:04:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T08:04:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:04:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T08:04:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap9 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap9 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9 2024-04-06T08:04:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:04:13.917 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T08:04:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T08:04:13.918 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T08:04:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:04:13.920+0000 7fdbb20e8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 32014) UID: 1000 2024-04-06T08:04:17.919 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T08:04:17.920 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:04:17.920 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T08:04:17.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T08:04:17.920 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T08:04:17.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T08:04:17.924 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-04-06T08:04:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:04:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:04:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:04:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:04:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:04:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:04:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:04:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:04:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:04:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:04:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:04:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:04:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:04:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:04:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:04:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:04:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:04:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:04:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:04:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:04:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:04:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:04:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:04:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:04:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:04:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:04:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:04:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:04:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:04:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:04:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:04:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:04:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:04:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:04:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:04:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:04:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:04:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:04:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:04:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:04:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:04:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:04:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:04:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:04:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:04:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 37 2024-04-06T08:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:04:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:04:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:04:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=37 2024-04-06T08:04:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:04:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:04:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:04:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:04:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:04:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:04:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:04:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:04:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:04:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:04:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:04:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 37 = '' 2024-04-06T08:04:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:04:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:04:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:04:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:04:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:04:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:04:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:04:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:04:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:04:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:04:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 37 = '' 2024-04-06T08:04:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:04:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:04:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:04:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:04:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:04:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:04:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 37 = '' 2024-04-06T08:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:04:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:04:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:04:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:04:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:04:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=37 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 37 = 37 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:04:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:04:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:04:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-04-06T08:04:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 18 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:02:58 2024 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:04:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:04:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:04:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:04:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:04:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:04:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap8-clean ']' 2024-04-06T08:04:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8-clean 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:04:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:04:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:04:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap9 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:04:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:04:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:04:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap9-clean 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9-clean 2024-04-06T08:04:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror test 2024-04-06T08:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:04:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:04:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:04:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-06T08:04:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 30613504 bytes 2024-04-06T08:04:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 134494208 bytes 2024-04-06T08:04:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 185792512 bytes 2024-04-06T08:04:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 218311680 bytes 2024-04-06T08:04:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 253316096 bytes 2024-04-06T08:04:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 308806656 bytes 2024-04-06T08:04:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 371742720 bytes 2024-04-06T08:04:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 419274752 bytes 2024-04-06T08:04:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 436286464 bytes 2024-04-06T08:04:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 441029632 bytes 2024-04-06T08:04:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 488416256 bytes 2024-04-06T08:04:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 498172928 bytes 2024-04-06T08:04:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 580967424 bytes 2024-04-06T08:04:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 586310656 bytes 2024-04-06T08:04:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 641987584 bytes 2024-04-06T08:04:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 671204352 bytes 2024-04-06T08:04:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 716514304 bytes 2024-04-06T08:04:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 735094784 bytes 2024-04-06T08:04:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 792736768 bytes 2024-04-06T08:04:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 852789248 bytes 2024-04-06T08:04:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 882279424 bytes 2024-04-06T08:04:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 899960832 bytes 2024-04-06T08:04:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 920397824 bytes 2024-04-06T08:04:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 972375040 bytes 2024-04-06T08:04:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1011842048 bytes 2024-04-06T08:04:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1012456448 bytes 2024-04-06T08:04:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1013076992 bytes 2024-04-06T08:04:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1064207360 bytes 2024-04-06T08:04:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1113629696 bytes 2024-04-06T08:04:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1158959104 bytes 2024-04-06T08:04:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1177888768 bytes 2024-04-06T08:04:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1238658048 bytes 2024-04-06T08:04:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1255562240 bytes 2024-04-06T08:04:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1292264448 bytes 2024-04-06T08:04:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1330468864 bytes 2024-04-06T08:04:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1367929856 bytes 2024-04-06T08:04:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1368553472 bytes 2024-04-06T08:04:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1416339456 bytes 2024-04-06T08:04:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1419607040 bytes 2024-04-06T08:04:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1486559232 bytes 2024-04-06T08:04:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1505124352 bytes 2024-04-06T08:04:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1549415424 bytes 2024-04-06T08:04:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1586841600 bytes 2024-04-06T08:04:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1661217792 bytes 2024-04-06T08:04:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1722692608 bytes 2024-04-06T08:04:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1786101760 bytes 2024-04-06T08:04:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1830364160 bytes 2024-04-06T08:04:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1867330560 bytes 2024-04-06T08:04:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1889782784 bytes 2024-04-06T08:04:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1949973504 bytes 2024-04-06T08:04:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1994280960 bytes 2024-04-06T08:04:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2054170624 bytes 2024-04-06T08:04:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2111472640 bytes 2024-04-06T08:04:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2182750208 bytes 2024-04-06T08:04:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2237526016 bytes 2024-04-06T08:04:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2297720832 bytes 2024-04-06T08:04:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2408311808 bytes 2024-04-06T08:04:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2442181632 bytes 2024-04-06T08:04:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2488967168 bytes 2024-04-06T08:04:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2490143744 bytes 2024-04-06T08:04:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2529550336 bytes 2024-04-06T08:04:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2576919552 bytes 2024-04-06T08:04:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2610573312 bytes 2024-04-06T08:04:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2627167232 bytes 2024-04-06T08:04:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2669575168 bytes 2024-04-06T08:04:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2684821504 bytes 2024-04-06T08:04:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2747934720 bytes 2024-04-06T08:04:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2748524544 bytes 2024-04-06T08:04:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2768618496 bytes 2024-04-06T08:04:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2786540544 bytes 2024-04-06T08:04:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2849924096 bytes 2024-04-06T08:04:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2850535424 bytes 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:04:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:04:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:04:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:04:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-06T08:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local description_pattern= 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local service_pattern= 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:04:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-06T08:04:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='test: 2024-04-06T08:04:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: global_id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:04:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+replaying 2024-04-06T08:04:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: replaying, {"bytes_per_second":18214594.21,"bytes_per_snapshot":528223232.0,"last_snapshot_bytes":528223232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390673,"remote_snapshot_timestamp":1712390673,"replay_state":"idle"} 2024-04-06T08:04:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: service: mirror.1 on smithi129 2024-04-06T08:04:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:04:44 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: peer_sites: 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: name: cluster2 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: state: up+stopped 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: description: local image is primary 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: last_update: 2024-04-06 08:04:44' 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ state: .*up+replaying' 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: state: up+replaying 2024-04-06T08:04:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^ description: .*' 2024-04-06T08:04:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: description: replaying, {"bytes_per_second":18214594.21,"bytes_per_snapshot":528223232.0,"last_snapshot_bytes":528223232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390673,"remote_snapshot_timestamp":1712390673,"replay_state":"idle"} 2024-04-06T08:04:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:04:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:04:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep '^up+' 2024-04-06T08:04:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:04:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep 'service: *mirror..* on ' 2024-04-06T08:04:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: service: mirror.1 on smithi129 2024-04-06T08:04:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update 2024-04-06T08:04:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:04:44' 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-06 08:04:44' 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state_pattern=up+replaying 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local 'prev_last_update=2024-04-06 08:04:44' 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ CEPH_ARGS= 2024-04-06T08:04:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-06T08:04:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status='OKOKOK142144269mirror.1smithi12918.2.2-1059-g4d05f4catrueOK43154318mirror.2smithi12918.2.2-1059-g4d05f4cafalseOK44734476mirror.3smithi12918.2.2-1059-g4d05f4cafalseOK45924595mirror.0smithi12918.2.2-1059-g4d05f4cafalseOKtestaedbdd61-8264-4828-b99b-205286b966e8up+replayingreplaying, {"bytes_per_second":18214594.21,"bytes_per_snapshot":528223232.0,"last_snapshot_bytes":528223232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1712390673,"remote_snapshot_timestamp":1712390673,"replay_state":"idle"}42144269mirror.1smithi1292024-04-06 08:04:44cluster23aea22be-a2a5-4341-813f-81096758e7a9up+stoppedlocal image is primary2024-04-06 08:04:44' 2024-04-06T08:04:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_update state 2024-04-06T08:04:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-06T08:04:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ last_update='2024-04-06 08:04:44' 2024-04-06T08:04:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-06T08:04:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=up+replaying 2024-04-06T08:04:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo up+replaying 2024-04-06T08:04:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep up+replaying 2024-04-06T08:04:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:up+replaying 2024-04-06T08:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap10 2024-04-06T08:04:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap10 2024-04-06T08:04:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:04:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:04:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:04:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10 2024-04-06T08:04:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:05:01.926 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9250861638423645) > weight (0.25) 2024-04-06T08:05:01.926 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7512376835880563) > weight (0.25) 2024-04-06T08:05:01.926 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7566341779290109) > weight (0.25) 2024-04-06T08:05:01.926 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-06T08:05:01.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-06T08:05:01.927 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T08:05:01.927 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-06T08:05:01.927 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:05:01.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:05:01.928+0000 7f19578e0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 28970) UID: 1000 2024-04-06T08:05:08.029 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-06T08:05:08.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T08:05:08.030 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T08:05:08.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T08:05:08.033 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:05:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:05:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:05:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:05:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:05:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:05:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:05:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:05:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:05:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:05:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:05:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:05:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:05:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:05:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:05:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:05:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:05:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:05:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:05:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:05:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:05:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:05:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:05:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:05:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:05:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:05:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:05:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:05:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:05:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:05:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:05:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:05:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:05:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:05:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:05:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:05:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:05:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:05:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:05:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:05:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:05:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:05:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:05:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:05:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:05:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:05:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:05:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:05:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:05:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:05:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:05:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:05:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:05:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 41 2024-04-06T08:05:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:05:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:05:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:05:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:05:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=41 2024-04-06T08:05:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:05:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:05:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:05:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:05:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:05:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:05:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:05:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:05:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:05:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:05:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 41 = '' 2024-04-06T08:05:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:05:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:05:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:05:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:05:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:05:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:05:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:05:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:05:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:05:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 41 = '' 2024-04-06T08:05:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:05:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:05:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:05:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:05:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:05:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:05:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:05:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:05:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:05:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 41 = '' 2024-04-06T08:05:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:05:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:05:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:05:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=41 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 41 = 41 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:05:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:05:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 20 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:05:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:04:39 2024 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:05:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:05:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snap9-clean ']' 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9-clean 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap10 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ clean_snap_name=snap10-clean 2024-04-06T08:05:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-04-06T08:05:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10-clean 2024-04-06T08:05:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:05:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:05:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:05:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:05:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-06T08:05:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:05:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:05:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:05:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:05:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:05:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:05:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:05:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:05:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:05:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:05:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:05:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:05:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:05:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:05:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:05:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:05:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:05:41.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:05:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:05:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-test.status 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-test.status 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-test.status 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-test.status 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-06T08:05:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 44 2024-04-06T08:05:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:05:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:05:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.status 2024-04-06T08:05:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-06T08:05:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:05:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=44 2024-04-06T08:05:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:05:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:05:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:05:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.status 2024-04-06T08:05:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-06T08:05:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:05:43.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=44 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 44 = 44 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10-clean 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:05:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:05:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'test': 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 21 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10a644ac003d 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10a644ac003d 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:05:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:04:39 2024 2024-04-06T08:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 07:55:47 2024 2024-04-06T08:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: aedbdd61-8264-4828-b99b-205286b966e8 2024-04-06T08:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:05:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 10 2024-04-06T08:05:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:05:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap1 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap1 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap1-clean 2024-04-06T08:05:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap1-clean 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:47.924 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T08:05:47.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T08:05:47.925 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T08:05:47.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:05:47.924+0000 7f2031916640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 34467) UID: 1000 2024-04-06T08:05:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:05:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap2 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap2 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap2-clean 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap2-clean 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:52.926 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T08:05:52.926 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:05:52.926 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T08:05:52.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T08:05:52.927 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T08:05:52.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T08:05:52.930 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-04-06T08:05:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap3 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap3 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap3-clean 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap3-clean 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap4 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap4 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:05:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:05:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:05:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap4-clean 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap4-clean 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap5 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap5 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap5-clean 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap5-clean 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap6 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap6 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap6-clean 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap6-clean 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap7 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap7 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap7-clean 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap7-clean 2024-04-06T08:06:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap8 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap8 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap8-clean 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap8-clean 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap9 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap9 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap9-clean 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap9-clean 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap10 2024-04-06T08:06:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap10 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap10-clean 2024-04-06T08:06:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror test snap10-clean 2024-04-06T08:06:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap10-clean 2024-04-06T08:06:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:06:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export 2024-04-06T08:06:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-06T08:06:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-test.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-test.export 2024-04-06T08:06:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 10 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap1 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap1 2024-04-06T08:06:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-04-06T08:06:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap1-clean 2024-04-06T08:06:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-04-06T08:06:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap1-clean 2024-04-06T08:06:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-04-06T08:06:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap2 2024-04-06T08:06:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-04-06T08:06:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap2 2024-04-06T08:06:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-04-06T08:06:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap2-clean 2024-04-06T08:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-04-06T08:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap2-clean 2024-04-06T08:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-04-06T08:06:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap3 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap3 2024-04-06T08:06:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-04-06T08:06:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap3-clean 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap3-clean 2024-04-06T08:06:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-04-06T08:06:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap4 2024-04-06T08:06:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-04-06T08:06:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap4 2024-04-06T08:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-04-06T08:06:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap4-clean 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap4-clean 2024-04-06T08:06:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-04-06T08:06:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap5 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap5 2024-04-06T08:06:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-04-06T08:06:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap5-clean 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap5-clean 2024-04-06T08:06:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-04-06T08:06:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap6 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap6 2024-04-06T08:06:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-04-06T08:06:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap6-clean 2024-04-06T08:06:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-04-06T08:06:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap6-clean 2024-04-06T08:06:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-04-06T08:06:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap7 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap7 2024-04-06T08:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-04-06T08:06:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap7-clean 2024-04-06T08:06:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-04-06T08:06:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap7-clean 2024-04-06T08:06:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-04-06T08:06:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap8 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap8 2024-04-06T08:06:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-04-06T08:06:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap8-clean 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap8-clean 2024-04-06T08:06:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-04-06T08:06:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap9 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap9 2024-04-06T08:06:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-04-06T08:06:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap9-clean 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap9-clean 2024-04-06T08:06:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-04-06T08:06:49.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7553317357050492) > weight (0.25) 2024-04-06T08:06:49.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41720460671179005) > weight (0.25) 2024-04-06T08:06:49.034 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T08:06:49.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T08:06:49.035 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T08:06:49.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:06:49.035+0000 7f5ca4301640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 33165) UID: 1000 2024-04-06T08:06:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap10 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap10 2024-04-06T08:06:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-04-06T08:06:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap10-clean 2024-04-06T08:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-04-06T08:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap10-clean 2024-04-06T08:06:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-04-06T08:06:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:06:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror test 2024-04-06T08:06:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:06:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:06:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror test 2024-04-06T08:06:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-04-06T08:06:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-04-06T08:06:51.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:06:51.403+0000 7f2f7a720640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x564ed866a4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:06:51.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:06:51.403+0000 7f2f80f2d640 -1 rbd::mirror::ImageReplayer: 0x564ed784aa00 [3/aedbdd61-8264-4828-b99b-205286b966e8] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:06:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:06:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror test 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=test 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-06T08:06:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:06:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-06T08:06:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:06:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:06:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:06:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=test 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror test 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:06:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=test 2024-04-06T08:06:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-06T08:06:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:06:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: create many images' 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T08:06:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T08:06:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 08:06:52 TEST: create many images 2024-04-06T08:06:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 08:06:52 TEST: create many images 2024-04-06T08:06:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ snap_name=snap 2024-04-06T08:06:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 50 2024-04-06T08:06:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:06:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_1 2024-04-06T08:06:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-04-06T08:06:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_1 128M 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:06:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:06:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:06:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:06:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:06:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:06:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-04-06T08:06:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:06:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-06T08:06:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_1': 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12708ed3a947 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12708ed3a947 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:06:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:06:52 2024 2024-04-06T08:06:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:06:52 2024 2024-04-06T08:06:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:06:52 2024 2024-04-06T08:06:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:06:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:06:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: eb012d1c-4f5e-44aa-9edc-096c064d4f16 2024-04-06T08:06:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-06T08:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:06:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:06:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:06:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:06:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 38555648 bytes 2024-04-06T08:06:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 70344704 bytes 2024-04-06T08:06:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 132414464 bytes 2024-04-06T08:06:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 194994176 bytes 2024-04-06T08:06:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 206578688 bytes 2024-04-06T08:06:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 253172736 bytes 2024-04-06T08:06:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 281036800 bytes 2024-04-06T08:06:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 378469376 bytes 2024-04-06T08:06:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 433553408 bytes 2024-04-06T08:06:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 480133120 bytes 2024-04-06T08:06:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 523107328 bytes 2024-04-06T08:06:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 563673088 bytes 2024-04-06T08:06:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 579723264 bytes 2024-04-06T08:06:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 588812288 bytes 2024-04-06T08:06:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 612751360 bytes 2024-04-06T08:06:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 647877632 bytes 2024-04-06T08:06:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 658661376 bytes 2024-04-06T08:06:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 679484416 bytes 2024-04-06T08:06:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 698559488 bytes 2024-04-06T08:06:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 730611712 bytes 2024-04-06T08:06:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 735726592 bytes 2024-04-06T08:06:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 810137600 bytes 2024-04-06T08:06:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 877841408 bytes 2024-04-06T08:06:57.036 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T08:06:57.036 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:06:57.036 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T08:06:57.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T08:06:57.037 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T08:06:57.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T08:06:57.039 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-04-06T08:06:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 886114304 bytes 2024-04-06T08:06:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 922779648 bytes 2024-04-06T08:06:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 970756096 bytes 2024-04-06T08:06:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1000792064 bytes 2024-04-06T08:06:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1071939584 bytes 2024-04-06T08:06:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1103662080 bytes 2024-04-06T08:06:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1136291840 bytes 2024-04-06T08:06:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1148552192 bytes 2024-04-06T08:06:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1259003904 bytes 2024-04-06T08:06:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1300321280 bytes 2024-04-06T08:06:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1300927488 bytes 2024-04-06T08:06:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1324122112 bytes 2024-04-06T08:06:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1360000000 bytes 2024-04-06T08:06:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1388817408 bytes 2024-04-06T08:06:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1433237504 bytes 2024-04-06T08:06:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1474661376 bytes 2024-04-06T08:06:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1500197888 bytes 2024-04-06T08:06:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1552562176 bytes 2024-04-06T08:07:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1599278080 bytes 2024-04-06T08:07:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1653349376 bytes 2024-04-06T08:07:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1698273280 bytes 2024-04-06T08:07:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1734373376 bytes 2024-04-06T08:07:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1810076672 bytes 2024-04-06T08:07:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1918416896 bytes 2024-04-06T08:07:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1926104064 bytes 2024-04-06T08:07:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1954013184 bytes 2024-04-06T08:07:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2011476992 bytes 2024-04-06T08:07:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2042738688 bytes 2024-04-06T08:07:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2056146944 bytes 2024-04-06T08:07:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2118558720 bytes 2024-04-06T08:07:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2174601216 bytes 2024-04-06T08:07:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2193446912 bytes 2024-04-06T08:07:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2235963392 bytes 2024-04-06T08:07:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2275697664 bytes 2024-04-06T08:07:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2390212608 bytes 2024-04-06T08:07:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2438212608 bytes 2024-04-06T08:07:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2467404800 bytes 2024-04-06T08:07:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2521816064 bytes 2024-04-06T08:07:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2562272256 bytes 2024-04-06T08:07:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2667800576 bytes 2024-04-06T08:07:04.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2685460480 bytes 2024-04-06T08:07:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2717555712 bytes 2024-04-06T08:07:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2795160576 bytes 2024-04-06T08:07:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2803170304 bytes 2024-04-06T08:07:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2860555264 bytes 2024-04-06T08:07:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2895344640 bytes 2024-04-06T08:07:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2961233920 bytes 2024-04-06T08:07:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2999862272 bytes 2024-04-06T08:07:05.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 3084269568 bytes 2024-04-06T08:07:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 3106399232 bytes 2024-04-06T08:07:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 3127511040 bytes 2024-04-06T08:07:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 3140230144 bytes 2024-04-06T08:07:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 3155316736 bytes 2024-04-06T08:07:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 3203694592 bytes 2024-04-06T08:07:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 3248741376 bytes 2024-04-06T08:07:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7900 IOs, 3369417728 bytes 2024-04-06T08:07:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8000 IOs, 3419747328 bytes 2024-04-06T08:07:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8100 IOs, 3506821120 bytes 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_2 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-04-06T08:07:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_2 128M 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2024-04-06T08:07:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:07:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-04-06T08:07:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:07:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-04-06T08:07:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:07:08.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_2': 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128254b566ee 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128254b566ee 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:07:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:07 2024 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:07 2024 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:07 2024 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 594aad0b-58bb-4181-a369-5cf04943564e 2024-04-06T08:07:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:07:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:07:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-06T08:07:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:07:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:07:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=5 2024-04-06T08:07:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:07:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:07:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 51890176 bytes 2024-04-06T08:07:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 75750400 bytes 2024-04-06T08:07:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 81480704 bytes 2024-04-06T08:07:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 147581952 bytes 2024-04-06T08:07:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 148206592 bytes 2024-04-06T08:07:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 178381824 bytes 2024-04-06T08:07:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 186387456 bytes 2024-04-06T08:07:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 249056256 bytes 2024-04-06T08:07:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 289753088 bytes 2024-04-06T08:07:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 300856320 bytes 2024-04-06T08:07:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 356215808 bytes 2024-04-06T08:07:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 409624576 bytes 2024-04-06T08:07:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 439695360 bytes 2024-04-06T08:07:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 462418944 bytes 2024-04-06T08:07:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 514880512 bytes 2024-04-06T08:07:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 590092288 bytes 2024-04-06T08:07:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 626025472 bytes 2024-04-06T08:07:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 667544576 bytes 2024-04-06T08:07:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 698265600 bytes 2024-04-06T08:07:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 722091008 bytes 2024-04-06T08:07:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 740096000 bytes 2024-04-06T08:07:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 796006400 bytes 2024-04-06T08:07:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 806520832 bytes 2024-04-06T08:07:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 832270336 bytes 2024-04-06T08:07:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 924364800 bytes 2024-04-06T08:07:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 948163584 bytes 2024-04-06T08:07:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 956638208 bytes 2024-04-06T08:07:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 999618560 bytes 2024-04-06T08:07:12.931 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7093659020472437) > weight (0.25) 2024-04-06T08:07:12.931 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-06T08:07:12.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-06T08:07:12.932 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-06T08:07:12.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:07:12.931+0000 7f39cfc1d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 28937) UID: 1000 2024-04-06T08:07:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1133734912 bytes 2024-04-06T08:07:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1147261952 bytes 2024-04-06T08:07:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1147870208 bytes 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_3 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_3 128M 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:07:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:07:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:07:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-04-06T08:07:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:07:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_3': 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128e601d8f22 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128e601d8f22 2024-04-06T08:07:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:13 2024 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:13 2024 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:13 2024 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 23b023dd-adbb-4aae-bbd9-c320028b0492 2024-04-06T08:07:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:07:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:07:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-06T08:07:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:07:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:07:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=10 2024-04-06T08:07:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:07:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:07:13.933 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-06T08:07:13.933 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:07:13.933 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-06T08:07:13.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T08:07:13.933 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T08:07:13.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T08:07:13.936 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-04-06T08:07:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 29097984 bytes 2024-04-06T08:07:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 102965248 bytes 2024-04-06T08:07:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 126383104 bytes 2024-04-06T08:07:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 205299712 bytes 2024-04-06T08:07:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 210534400 bytes 2024-04-06T08:07:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 247206912 bytes 2024-04-06T08:07:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 273480704 bytes 2024-04-06T08:07:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 311281664 bytes 2024-04-06T08:07:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 353992704 bytes 2024-04-06T08:07:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 378321920 bytes 2024-04-06T08:07:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 396191744 bytes 2024-04-06T08:07:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 443796480 bytes 2024-04-06T08:07:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 462995456 bytes 2024-04-06T08:07:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 467878912 bytes 2024-04-06T08:07:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 511163392 bytes 2024-04-06T08:07:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 521279488 bytes 2024-04-06T08:07:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 545989632 bytes 2024-04-06T08:07:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 558310400 bytes 2024-04-06T08:07:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 575344640 bytes 2024-04-06T08:07:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 604207104 bytes 2024-04-06T08:07:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 652309504 bytes 2024-04-06T08:07:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 704449536 bytes 2024-04-06T08:07:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 708945920 bytes 2024-04-06T08:07:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 724272128 bytes 2024-04-06T08:07:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 732399616 bytes 2024-04-06T08:07:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 748470272 bytes 2024-04-06T08:07:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 779942912 bytes 2024-04-06T08:07:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 846573568 bytes 2024-04-06T08:07:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 862441472 bytes 2024-04-06T08:07:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 883666944 bytes 2024-04-06T08:07:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 923758592 bytes 2024-04-06T08:07:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1005588480 bytes 2024-04-06T08:07:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1077690368 bytes 2024-04-06T08:07:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1111979008 bytes 2024-04-06T08:07:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1165795328 bytes 2024-04-06T08:07:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1213540352 bytes 2024-04-06T08:07:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1238404096 bytes 2024-04-06T08:07:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1291111424 bytes 2024-04-06T08:07:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1327143936 bytes 2024-04-06T08:07:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1350667264 bytes 2024-04-06T08:07:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1386533888 bytes 2024-04-06T08:07:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1430562816 bytes 2024-04-06T08:07:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1502409728 bytes 2024-04-06T08:07:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1558121472 bytes 2024-04-06T08:07:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1586250752 bytes 2024-04-06T08:07:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1589804032 bytes 2024-04-06T08:07:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1617329152 bytes 2024-04-06T08:07:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1679682560 bytes 2024-04-06T08:07:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1798380544 bytes 2024-04-06T08:07:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1940804608 bytes 2024-04-06T08:07:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1976522752 bytes 2024-04-06T08:07:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2056711168 bytes 2024-04-06T08:07:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2115066880 bytes 2024-04-06T08:07:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2134064128 bytes 2024-04-06T08:07:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2194286592 bytes 2024-04-06T08:07:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2202558464 bytes 2024-04-06T08:07:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2239056896 bytes 2024-04-06T08:07:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2276160512 bytes 2024-04-06T08:07:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2309858304 bytes 2024-04-06T08:07:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2348781568 bytes 2024-04-06T08:07:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_4 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:07:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_4 128M 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:07:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2024-04-06T08:07:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:07:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-04-06T08:07:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:07:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-04-06T08:07:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:07:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-06T08:07:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_4': 2024-04-06T08:07:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:07:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:07:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12a074cbb4bf 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12a074cbb4bf 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:23 2024 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:23 2024 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:23 2024 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ca09eaed-79f4-4ee1-8861-740c13842056 2024-04-06T08:07:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:07:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:07:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-06T08:07:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:07:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:07:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:07:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:07:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:07:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 75599872 bytes 2024-04-06T08:07:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 131198976 bytes 2024-04-06T08:07:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 208956416 bytes 2024-04-06T08:07:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 260180992 bytes 2024-04-06T08:07:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 311802880 bytes 2024-04-06T08:07:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 325331968 bytes 2024-04-06T08:07:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 348965888 bytes 2024-04-06T08:07:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 363661312 bytes 2024-04-06T08:07:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 418409472 bytes 2024-04-06T08:07:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 462556160 bytes 2024-04-06T08:07:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 505374720 bytes 2024-04-06T08:07:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 505978880 bytes 2024-04-06T08:07:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 573999104 bytes 2024-04-06T08:07:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 638286848 bytes 2024-04-06T08:07:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 686790656 bytes 2024-04-06T08:07:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 743777280 bytes 2024-04-06T08:07:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 795799552 bytes 2024-04-06T08:07:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 828866560 bytes 2024-04-06T08:07:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 851129344 bytes 2024-04-06T08:07:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 868842496 bytes 2024-04-06T08:07:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 924440576 bytes 2024-04-06T08:07:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 996407296 bytes 2024-04-06T08:07:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1023198208 bytes 2024-04-06T08:07:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1075750912 bytes 2024-04-06T08:07:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1144743936 bytes 2024-04-06T08:07:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1162360832 bytes 2024-04-06T08:07:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1226690560 bytes 2024-04-06T08:07:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1273270272 bytes 2024-04-06T08:07:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1306374144 bytes 2024-04-06T08:07:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1339207680 bytes 2024-04-06T08:07:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1372063744 bytes 2024-04-06T08:07:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1392082944 bytes 2024-04-06T08:07:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1438101504 bytes 2024-04-06T08:07:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1514853376 bytes 2024-04-06T08:07:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1553913856 bytes 2024-04-06T08:07:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1579979776 bytes 2024-04-06T08:07:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1603065856 bytes 2024-04-06T08:07:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1661336576 bytes 2024-04-06T08:07:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1671272448 bytes 2024-04-06T08:07:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1703376896 bytes 2024-04-06T08:07:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1800239104 bytes 2024-04-06T08:07:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1819825152 bytes 2024-04-06T08:07:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1852941312 bytes 2024-04-06T08:07:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1873886208 bytes 2024-04-06T08:07:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1923166208 bytes 2024-04-06T08:07:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1962314752 bytes 2024-04-06T08:07:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2029417472 bytes 2024-04-06T08:07:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2030020608 bytes 2024-04-06T08:07:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2098179072 bytes 2024-04-06T08:07:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2186038272 bytes 2024-04-06T08:07:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2186642432 bytes 2024-04-06T08:07:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2215090176 bytes 2024-04-06T08:07:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2237061120 bytes 2024-04-06T08:07:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2262666240 bytes 2024-04-06T08:07:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2282290176 bytes 2024-04-06T08:07:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2323975168 bytes 2024-04-06T08:07:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2375764992 bytes 2024-04-06T08:07:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2396766208 bytes 2024-04-06T08:07:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2425363456 bytes 2024-04-06T08:07:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2481024000 bytes 2024-04-06T08:07:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2487204864 bytes 2024-04-06T08:07:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2518109184 bytes 2024-04-06T08:07:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2564152320 bytes 2024-04-06T08:07:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2599400448 bytes 2024-04-06T08:07:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2638790656 bytes 2024-04-06T08:07:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2639385600 bytes 2024-04-06T08:07:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2673256448 bytes 2024-04-06T08:07:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2727596032 bytes 2024-04-06T08:07:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2818589696 bytes 2024-04-06T08:07:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2875973632 bytes 2024-04-06T08:07:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2900734976 bytes 2024-04-06T08:07:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2978272256 bytes 2024-04-06T08:07:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 3010354176 bytes 2024-04-06T08:07:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 3078246400 bytes 2024-04-06T08:07:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 3118370816 bytes 2024-04-06T08:07:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 3163630592 bytes 2024-04-06T08:07:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 3202667520 bytes 2024-04-06T08:07:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 3258420224 bytes 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_5 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-04-06T08:07:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_5 128M 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2024-04-06T08:07:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:07:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-04-06T08:07:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:07:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-04-06T08:07:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:07:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-06T08:07:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_5': 2024-04-06T08:07:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:07:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12a96d0905bf 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12a96d0905bf 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:38 2024 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:38 2024 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:38 2024 2024-04-06T08:07:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:07:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:07:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: b950a79f-80c3-4891-a04f-2ec8f28298d1 2024-04-06T08:07:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:07:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:07:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-06T08:07:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:07:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:07:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:07:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:07:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 11714560 bytes 2024-04-06T08:07:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 40163328 bytes 2024-04-06T08:07:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 121560064 bytes 2024-04-06T08:07:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 122166272 bytes 2024-04-06T08:07:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 136284160 bytes 2024-04-06T08:07:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 172576768 bytes 2024-04-06T08:07:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 225277952 bytes 2024-04-06T08:07:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 247998464 bytes 2024-04-06T08:07:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 285039616 bytes 2024-04-06T08:07:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 337492992 bytes 2024-04-06T08:07:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 400949248 bytes 2024-04-06T08:07:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 491865088 bytes 2024-04-06T08:07:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 515144704 bytes 2024-04-06T08:07:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 552452096 bytes 2024-04-06T08:07:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 571413504 bytes 2024-04-06T08:07:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 604119040 bytes 2024-04-06T08:07:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 615758848 bytes 2024-04-06T08:07:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 632804352 bytes 2024-04-06T08:07:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 655068160 bytes 2024-04-06T08:07:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 726909952 bytes 2024-04-06T08:07:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 796508160 bytes 2024-04-06T08:07:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 843634688 bytes 2024-04-06T08:07:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 891870208 bytes 2024-04-06T08:07:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 916982784 bytes 2024-04-06T08:07:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 949005312 bytes 2024-04-06T08:07:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 949632000 bytes 2024-04-06T08:07:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1009041408 bytes 2024-04-06T08:07:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1034236928 bytes 2024-04-06T08:07:44.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1072106496 bytes 2024-04-06T08:07:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1088542720 bytes 2024-04-06T08:07:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1149572096 bytes 2024-04-06T08:07:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1184362496 bytes 2024-04-06T08:07:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1213097984 bytes 2024-04-06T08:07:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1263220736 bytes 2024-04-06T08:07:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1280045056 bytes 2024-04-06T08:07:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1352368128 bytes 2024-04-06T08:07:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1385194496 bytes 2024-04-06T08:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1473963008 bytes 2024-04-06T08:07:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1518551040 bytes 2024-04-06T08:07:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1588405248 bytes 2024-04-06T08:07:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1625998336 bytes 2024-04-06T08:07:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1668305920 bytes 2024-04-06T08:07:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1686081536 bytes 2024-04-06T08:07:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1693394944 bytes 2024-04-06T08:07:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1765964800 bytes 2024-04-06T08:07:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1825545216 bytes 2024-04-06T08:07:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1871674368 bytes 2024-04-06T08:07:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1917372416 bytes 2024-04-06T08:07:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1948178432 bytes 2024-04-06T08:07:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1948784640 bytes 2024-04-06T08:07:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1986003968 bytes 2024-04-06T08:07:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2053039104 bytes 2024-04-06T08:07:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2092978176 bytes 2024-04-06T08:07:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2142660608 bytes 2024-04-06T08:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2193193984 bytes 2024-04-06T08:07:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2223558656 bytes 2024-04-06T08:07:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2224175104 bytes 2024-04-06T08:07:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2287560704 bytes 2024-04-06T08:07:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2358377472 bytes 2024-04-06T08:07:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2401748992 bytes 2024-04-06T08:07:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2402363392 bytes 2024-04-06T08:07:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2447891456 bytes 2024-04-06T08:07:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2514125824 bytes 2024-04-06T08:07:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2514743296 bytes 2024-04-06T08:07:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2536207360 bytes 2024-04-06T08:07:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2599434240 bytes 2024-04-06T08:07:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2620537856 bytes 2024-04-06T08:07:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2637987840 bytes 2024-04-06T08:07:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2682057728 bytes 2024-04-06T08:07:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2716095488 bytes 2024-04-06T08:07:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2784964608 bytes 2024-04-06T08:07:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2808534016 bytes 2024-04-06T08:07:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 2857974784 bytes 2024-04-06T08:07:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 2862298112 bytes 2024-04-06T08:07:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 2909959168 bytes 2024-04-06T08:07:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 2912433152 bytes 2024-04-06T08:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 2931677184 bytes 2024-04-06T08:07:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 2967348224 bytes 2024-04-06T08:07:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7900 IOs, 3002676224 bytes 2024-04-06T08:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:07:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_6 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_6 128M 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:07:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:07:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-04-06T08:07:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:07:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_6': 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12b16895666a 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12b16895666a 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:07:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:52 2024 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:52 2024 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:52 2024 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 6347dabe-e24c-40dc-91f2-8040b6675422 2024-04-06T08:07:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:07:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:07:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-06T08:07:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:07:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:07:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:07:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:07:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:07:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:07:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:07:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 12989440 bytes 2024-04-06T08:07:53.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 30415872 bytes 2024-04-06T08:07:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 69735424 bytes 2024-04-06T08:07:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 84646912 bytes 2024-04-06T08:07:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 131083264 bytes 2024-04-06T08:07:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 140949504 bytes 2024-04-06T08:07:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 221322240 bytes 2024-04-06T08:07:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 291079168 bytes 2024-04-06T08:07:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 355945472 bytes 2024-04-06T08:07:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 432545792 bytes 2024-04-06T08:07:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 486881280 bytes 2024-04-06T08:07:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 530176000 bytes 2024-04-06T08:07:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 565991424 bytes 2024-04-06T08:07:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 622915584 bytes 2024-04-06T08:07:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 667697152 bytes 2024-04-06T08:07:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 685515776 bytes 2024-04-06T08:07:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 709374976 bytes 2024-04-06T08:07:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 763724800 bytes 2024-04-06T08:07:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 796831744 bytes 2024-04-06T08:07:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 842991616 bytes 2024-04-06T08:07:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 882702336 bytes 2024-04-06T08:07:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 945364992 bytes 2024-04-06T08:07:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 986397696 bytes 2024-04-06T08:07:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1028583424 bytes 2024-04-06T08:07:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1098189824 bytes 2024-04-06T08:07:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1148838912 bytes 2024-04-06T08:07:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1177464832 bytes 2024-04-06T08:07:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1191593984 bytes 2024-04-06T08:07:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1227527168 bytes 2024-04-06T08:07:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1292889088 bytes 2024-04-06T08:07:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1306547200 bytes 2024-04-06T08:07:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1351536640 bytes 2024-04-06T08:07:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1425938432 bytes 2024-04-06T08:07:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1478528000 bytes 2024-04-06T08:07:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1531634688 bytes 2024-04-06T08:07:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1551590400 bytes 2024-04-06T08:07:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1575700480 bytes 2024-04-06T08:07:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1617577984 bytes 2024-04-06T08:07:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1628215296 bytes 2024-04-06T08:07:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1640051712 bytes 2024-04-06T08:07:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1681702912 bytes 2024-04-06T08:07:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1718716416 bytes 2024-04-06T08:08:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1777191936 bytes 2024-04-06T08:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1841410048 bytes 2024-04-06T08:08:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1927883776 bytes 2024-04-06T08:08:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1961560064 bytes 2024-04-06T08:08:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2053763072 bytes 2024-04-06T08:08:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2101453824 bytes 2024-04-06T08:08:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2107478016 bytes 2024-04-06T08:08:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2165329920 bytes 2024-04-06T08:08:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2202426368 bytes 2024-04-06T08:08:02.040 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8039816912250399) > weight (0.25) 2024-04-06T08:08:02.040 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.27645448837850806) > weight (0.25) 2024-04-06T08:08:02.040 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4398376985812581) > weight (0.25) 2024-04-06T08:08:02.040 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6265290041822411) > weight (0.25) 2024-04-06T08:08:02.040 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-04-06T08:08:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2263197696 bytes 2024-04-06T08:08:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2297819136 bytes 2024-04-06T08:08:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2359443456 bytes 2024-04-06T08:08:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2379839488 bytes 2024-04-06T08:08:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2403581952 bytes 2024-04-06T08:08:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2468118528 bytes 2024-04-06T08:08:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2514349056 bytes 2024-04-06T08:08:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2536086528 bytes 2024-04-06T08:08:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2548123648 bytes 2024-04-06T08:08:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2575005696 bytes 2024-04-06T08:08:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2600512512 bytes 2024-04-06T08:08:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2622193664 bytes 2024-04-06T08:08:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2658012160 bytes 2024-04-06T08:08:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2699214848 bytes 2024-04-06T08:08:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2779575296 bytes 2024-04-06T08:08:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2812199936 bytes 2024-04-06T08:08:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2897845248 bytes 2024-04-06T08:08:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2997786624 bytes 2024-04-06T08:08:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 3046450176 bytes 2024-04-06T08:08:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 3082892288 bytes 2024-04-06T08:08:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 3140132864 bytes 2024-04-06T08:08:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_7 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:08:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_7 128M 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2024-04-06T08:08:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:08:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-04-06T08:08:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:08:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-04-06T08:08:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_7': 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12b998551032 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12b998551032 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:08:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:05 2024 2024-04-06T08:08:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:05 2024 2024-04-06T08:08:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:05 2024 2024-04-06T08:08:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:08:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:08:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 839a8373-291e-4850-b111-46075edd9765 2024-04-06T08:08:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:08:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:08:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-06T08:08:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:08:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:08:06.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:08:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:08:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:08:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 24691712 bytes 2024-04-06T08:08:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 57873408 bytes 2024-04-06T08:08:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 86882304 bytes 2024-04-06T08:08:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 124247040 bytes 2024-04-06T08:08:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 147599360 bytes 2024-04-06T08:08:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 205404160 bytes 2024-04-06T08:08:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 241188864 bytes 2024-04-06T08:08:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 301202432 bytes 2024-04-06T08:08:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 323391488 bytes 2024-04-06T08:08:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 367444992 bytes 2024-04-06T08:08:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 447882240 bytes 2024-04-06T08:08:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 515691520 bytes 2024-04-06T08:08:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 543333376 bytes 2024-04-06T08:08:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 612361216 bytes 2024-04-06T08:08:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 707401728 bytes 2024-04-06T08:08:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 719031296 bytes 2024-04-06T08:08:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 727732224 bytes 2024-04-06T08:08:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 737323008 bytes 2024-04-06T08:08:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 762842112 bytes 2024-04-06T08:08:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 815176704 bytes 2024-04-06T08:08:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 845803520 bytes 2024-04-06T08:08:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 881355776 bytes 2024-04-06T08:08:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 936267776 bytes 2024-04-06T08:08:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1013547008 bytes 2024-04-06T08:08:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1091739648 bytes 2024-04-06T08:08:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1129714688 bytes 2024-04-06T08:08:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1197633536 bytes 2024-04-06T08:08:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1212399616 bytes 2024-04-06T08:08:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1310441472 bytes 2024-04-06T08:08:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1336004608 bytes 2024-04-06T08:08:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1364185088 bytes 2024-04-06T08:08:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1364805632 bytes 2024-04-06T08:08:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1417581568 bytes 2024-04-06T08:08:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1449991168 bytes 2024-04-06T08:08:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1467020288 bytes 2024-04-06T08:08:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1551104000 bytes 2024-04-06T08:08:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1649888256 bytes 2024-04-06T08:08:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1723586560 bytes 2024-04-06T08:08:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1803076608 bytes 2024-04-06T08:08:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1848839168 bytes 2024-04-06T08:08:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1910770688 bytes 2024-04-06T08:08:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1936586752 bytes 2024-04-06T08:08:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1943921664 bytes 2024-04-06T08:08:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2025008128 bytes 2024-04-06T08:08:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2126605312 bytes 2024-04-06T08:08:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2207711232 bytes 2024-04-06T08:08:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2240304128 bytes 2024-04-06T08:08:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2261417984 bytes 2024-04-06T08:08:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2330209280 bytes 2024-04-06T08:08:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_8 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_8 128M 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2024-04-06T08:08:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:08:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-04-06T08:08:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:08:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-04-06T08:08:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:08:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-06T08:08:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_8': 2024-04-06T08:08:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:08:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:08:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:08:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12c28889d099 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12c28889d099 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:15 2024 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:15 2024 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:15 2024 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: be4f8eb9-e09c-4224-856e-896364fa79d2 2024-04-06T08:08:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:08:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:08:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-06T08:08:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:08:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:08:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=11 2024-04-06T08:08:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:08:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:08:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 5745664 bytes 2024-04-06T08:08:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 86153216 bytes 2024-04-06T08:08:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 114842624 bytes 2024-04-06T08:08:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 115476480 bytes 2024-04-06T08:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 164211712 bytes 2024-04-06T08:08:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 198505472 bytes 2024-04-06T08:08:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 268875776 bytes 2024-04-06T08:08:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 319863808 bytes 2024-04-06T08:08:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 327375872 bytes 2024-04-06T08:08:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 371992576 bytes 2024-04-06T08:08:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 404305920 bytes 2024-04-06T08:08:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 420481024 bytes 2024-04-06T08:08:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 465069056 bytes 2024-04-06T08:08:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 506807296 bytes 2024-04-06T08:08:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 575013888 bytes 2024-04-06T08:08:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 688174080 bytes 2024-04-06T08:08:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 714078208 bytes 2024-04-06T08:08:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 723988480 bytes 2024-04-06T08:08:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 800267264 bytes 2024-04-06T08:08:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 841767936 bytes 2024-04-06T08:08:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 891164672 bytes 2024-04-06T08:08:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 943311872 bytes 2024-04-06T08:08:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 952901632 bytes 2024-04-06T08:08:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 962221056 bytes 2024-04-06T08:08:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1035862016 bytes 2024-04-06T08:08:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1039323136 bytes 2024-04-06T08:08:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1048323072 bytes 2024-04-06T08:08:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1067899904 bytes 2024-04-06T08:08:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1149070336 bytes 2024-04-06T08:08:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1258797056 bytes 2024-04-06T08:08:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1282167808 bytes 2024-04-06T08:08:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1318573056 bytes 2024-04-06T08:08:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1375280128 bytes 2024-04-06T08:08:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1448380416 bytes 2024-04-06T08:08:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1479892992 bytes 2024-04-06T08:08:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1518085120 bytes 2024-04-06T08:08:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1586743296 bytes 2024-04-06T08:08:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1619379200 bytes 2024-04-06T08:08:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1672056832 bytes 2024-04-06T08:08:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1694995456 bytes 2024-04-06T08:08:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1751282688 bytes 2024-04-06T08:08:23.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1785366528 bytes 2024-04-06T08:08:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1851516928 bytes 2024-04-06T08:08:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1882066944 bytes 2024-04-06T08:08:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1944323072 bytes 2024-04-06T08:08:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1957222400 bytes 2024-04-06T08:08:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1965650944 bytes 2024-04-06T08:08:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1989520384 bytes 2024-04-06T08:08:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2004586496 bytes 2024-04-06T08:08:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2072583168 bytes 2024-04-06T08:08:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2112567296 bytes 2024-04-06T08:08:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2148818944 bytes 2024-04-06T08:08:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2254922752 bytes 2024-04-06T08:08:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2281871360 bytes 2024-04-06T08:08:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2336487424 bytes 2024-04-06T08:08:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2364262400 bytes 2024-04-06T08:08:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2437177344 bytes 2024-04-06T08:08:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2476606464 bytes 2024-04-06T08:08:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2510494720 bytes 2024-04-06T08:08:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2511121408 bytes 2024-04-06T08:08:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2553521152 bytes 2024-04-06T08:08:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2569628672 bytes 2024-04-06T08:08:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2623758336 bytes 2024-04-06T08:08:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:08:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:08:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_9 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_9 128M 2024-04-06T08:08:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-04-06T08:08:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:08:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_9': 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12cbe348718f 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12cbe348718f 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:08:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:27 2024 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:27 2024 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:27 2024 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 760529a0-2277-4e6f-aa85-bf596b26afe0 2024-04-06T08:08:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:08:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:08:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-06T08:08:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:08:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:08:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:08:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:08:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:08:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 4473856 bytes 2024-04-06T08:08:27.936 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5491654149091814) > weight (0.25) 2024-04-06T08:08:27.937 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.388005359521323) > weight (0.25) 2024-04-06T08:08:27.937 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6709942897810258) > weight (0.25) 2024-04-06T08:08:27.937 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T08:08:27.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T08:08:27.937 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T08:08:27.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:08:27.937+0000 7f2daa066640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 32573) UID: 1000 2024-04-06T08:08:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 65771520 bytes 2024-04-06T08:08:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 180547584 bytes 2024-04-06T08:08:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 195333120 bytes 2024-04-06T08:08:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 238858240 bytes 2024-04-06T08:08:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 239465472 bytes 2024-04-06T08:08:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 241913856 bytes 2024-04-06T08:08:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 334024704 bytes 2024-04-06T08:08:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 362220544 bytes 2024-04-06T08:08:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 413298688 bytes 2024-04-06T08:08:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 436603904 bytes 2024-04-06T08:08:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 468072448 bytes 2024-04-06T08:08:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 470756352 bytes 2024-04-06T08:08:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 480266240 bytes 2024-04-06T08:08:30.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 524607488 bytes 2024-04-06T08:08:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 553055232 bytes 2024-04-06T08:08:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 593143808 bytes 2024-04-06T08:08:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 609294336 bytes 2024-04-06T08:08:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 650392576 bytes 2024-04-06T08:08:30.939 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T08:08:30.939 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:08:30.939 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T08:08:30.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T08:08:30.940 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T08:08:30.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T08:08:30.942 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-04-06T08:08:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 691531776 bytes 2024-04-06T08:08:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 727962624 bytes 2024-04-06T08:08:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 753004544 bytes 2024-04-06T08:08:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 778302464 bytes 2024-04-06T08:08:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 833359872 bytes 2024-04-06T08:08:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 905403392 bytes 2024-04-06T08:08:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 944654336 bytes 2024-04-06T08:08:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 955802624 bytes 2024-04-06T08:08:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 972260352 bytes 2024-04-06T08:08:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1032781824 bytes 2024-04-06T08:08:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1055209472 bytes 2024-04-06T08:08:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1069069312 bytes 2024-04-06T08:08:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1129736192 bytes 2024-04-06T08:08:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1180697600 bytes 2024-04-06T08:08:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1219451904 bytes 2024-04-06T08:08:33.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1276244992 bytes 2024-04-06T08:08:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1283588096 bytes 2024-04-06T08:08:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1319249920 bytes 2024-04-06T08:08:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1360278528 bytes 2024-04-06T08:08:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1399399424 bytes 2024-04-06T08:08:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1468138496 bytes 2024-04-06T08:08:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1503964160 bytes 2024-04-06T08:08:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1533410304 bytes 2024-04-06T08:08:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1591764992 bytes 2024-04-06T08:08:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1624903680 bytes 2024-04-06T08:08:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1640828928 bytes 2024-04-06T08:08:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1662695424 bytes 2024-04-06T08:08:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1692465152 bytes 2024-04-06T08:08:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1738716160 bytes 2024-04-06T08:08:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1845365760 bytes 2024-04-06T08:08:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1906795520 bytes 2024-04-06T08:08:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1932897280 bytes 2024-04-06T08:08:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1977974784 bytes 2024-04-06T08:08:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 1983086592 bytes 2024-04-06T08:08:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2038248448 bytes 2024-04-06T08:08:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2120589312 bytes 2024-04-06T08:08:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2134408192 bytes 2024-04-06T08:08:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2163692544 bytes 2024-04-06T08:08:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2195195904 bytes 2024-04-06T08:08:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2258721792 bytes 2024-04-06T08:08:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2287269888 bytes 2024-04-06T08:08:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2329017344 bytes 2024-04-06T08:08:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2420912128 bytes 2024-04-06T08:08:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2434206720 bytes 2024-04-06T08:08:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2464758784 bytes 2024-04-06T08:08:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2465355776 bytes 2024-04-06T08:08:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2557539328 bytes 2024-04-06T08:08:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2574203904 bytes 2024-04-06T08:08:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2631949312 bytes 2024-04-06T08:08:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2647337984 bytes 2024-04-06T08:08:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2695220224 bytes 2024-04-06T08:08:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2751512576 bytes 2024-04-06T08:08:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2766693376 bytes 2024-04-06T08:08:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 2803506176 bytes 2024-04-06T08:08:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 2892953600 bytes 2024-04-06T08:08:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 2909085696 bytes 2024-04-06T08:08:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:08:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:08:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:08:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_10 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_10 128M 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:40.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-04-06T08:08:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:08:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_10': 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12d9c91a3964 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12d9c91a3964 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:08:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:40 2024 2024-04-06T08:08:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:40 2024 2024-04-06T08:08:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:40 2024 2024-04-06T08:08:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:08:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:08:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 1626ab18-b923-4d9b-b871-b9490fbd81d7 2024-04-06T08:08:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:08:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:08:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-06T08:08:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:08:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:08:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=6 2024-04-06T08:08:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:08:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:08:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 81431552 bytes 2024-04-06T08:08:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 192869376 bytes 2024-04-06T08:08:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 247718912 bytes 2024-04-06T08:08:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 277956608 bytes 2024-04-06T08:08:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 323523584 bytes 2024-04-06T08:08:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 341621760 bytes 2024-04-06T08:08:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 419972096 bytes 2024-04-06T08:08:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 458268672 bytes 2024-04-06T08:08:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 521166848 bytes 2024-04-06T08:08:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 530131968 bytes 2024-04-06T08:08:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 621922304 bytes 2024-04-06T08:08:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 653204480 bytes 2024-04-06T08:08:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 691184640 bytes 2024-04-06T08:08:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 722100224 bytes 2024-04-06T08:08:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 774663168 bytes 2024-04-06T08:08:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 844709888 bytes 2024-04-06T08:08:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 900126720 bytes 2024-04-06T08:08:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 926156800 bytes 2024-04-06T08:08:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 965929984 bytes 2024-04-06T08:08:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 988420096 bytes 2024-04-06T08:08:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1034725376 bytes 2024-04-06T08:08:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1045284864 bytes 2024-04-06T08:08:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1097730048 bytes 2024-04-06T08:08:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1098364928 bytes 2024-04-06T08:08:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1152778240 bytes 2024-04-06T08:08:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1169014784 bytes 2024-04-06T08:08:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1195698176 bytes 2024-04-06T08:08:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1218739200 bytes 2024-04-06T08:08:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1265564672 bytes 2024-04-06T08:08:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1275892736 bytes 2024-04-06T08:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1322861568 bytes 2024-04-06T08:08:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1406468096 bytes 2024-04-06T08:08:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_11 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:08:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_11 128M 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-04-06T08:08:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:08:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_11': 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12e1f53db4b0 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12e1f53db4b0 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:08:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:47 2024 2024-04-06T08:08:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:47 2024 2024-04-06T08:08:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:47 2024 2024-04-06T08:08:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:08:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:08:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f7102142-b3e4-44ed-9508-3bf8555f6258 2024-04-06T08:08:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:08:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:08:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_11 2024-04-06T08:08:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:08:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:08:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=11 2024-04-06T08:08:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:08:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:08:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 59504640 bytes 2024-04-06T08:08:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 71919616 bytes 2024-04-06T08:08:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 98196480 bytes 2024-04-06T08:08:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 111048704 bytes 2024-04-06T08:08:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 190312448 bytes 2024-04-06T08:08:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 265969664 bytes 2024-04-06T08:08:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 299404288 bytes 2024-04-06T08:08:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 356190208 bytes 2024-04-06T08:08:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 389661696 bytes 2024-04-06T08:08:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 406494208 bytes 2024-04-06T08:08:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 432564224 bytes 2024-04-06T08:08:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 459524096 bytes 2024-04-06T08:08:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 483026944 bytes 2024-04-06T08:08:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 541244416 bytes 2024-04-06T08:08:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 580260864 bytes 2024-04-06T08:08:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 673055744 bytes 2024-04-06T08:08:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 732198912 bytes 2024-04-06T08:08:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 792646656 bytes 2024-04-06T08:08:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 829099008 bytes 2024-04-06T08:08:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 882850816 bytes 2024-04-06T08:08:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 948347904 bytes 2024-04-06T08:08:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 983425024 bytes 2024-04-06T08:08:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 997712896 bytes 2024-04-06T08:08:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1039132672 bytes 2024-04-06T08:08:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1087273984 bytes 2024-04-06T08:08:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1124830208 bytes 2024-04-06T08:08:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1204942848 bytes 2024-04-06T08:08:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1232018432 bytes 2024-04-06T08:08:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1275892736 bytes 2024-04-06T08:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1345187840 bytes 2024-04-06T08:08:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1365554176 bytes 2024-04-06T08:08:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1366182912 bytes 2024-04-06T08:08:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1420403712 bytes 2024-04-06T08:08:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1421000704 bytes 2024-04-06T08:08:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1434298368 bytes 2024-04-06T08:08:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1450748928 bytes 2024-04-06T08:08:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1499278336 bytes 2024-04-06T08:08:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1521540096 bytes 2024-04-06T08:08:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1531655168 bytes 2024-04-06T08:08:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1598451712 bytes 2024-04-06T08:08:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1659873280 bytes 2024-04-06T08:08:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1672450048 bytes 2024-04-06T08:08:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1679983616 bytes 2024-04-06T08:08:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1756654592 bytes 2024-04-06T08:08:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1812602880 bytes 2024-04-06T08:08:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1848675328 bytes 2024-04-06T08:08:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1904627712 bytes 2024-04-06T08:08:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1913724928 bytes 2024-04-06T08:08:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1951505408 bytes 2024-04-06T08:08:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2025724928 bytes 2024-04-06T08:08:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2037446656 bytes 2024-04-06T08:08:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2092921856 bytes 2024-04-06T08:08:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2101870592 bytes 2024-04-06T08:08:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2145903616 bytes 2024-04-06T08:08:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2203771904 bytes 2024-04-06T08:08:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2238404608 bytes 2024-04-06T08:08:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2286529536 bytes 2024-04-06T08:08:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2299812864 bytes 2024-04-06T08:08:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2363444224 bytes 2024-04-06T08:08:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2419857408 bytes 2024-04-06T08:08:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2452495360 bytes 2024-04-06T08:08:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2511209472 bytes 2024-04-06T08:08:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2601550848 bytes 2024-04-06T08:08:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2634773504 bytes 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_12 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_12 128M 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:08:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:08:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2024-04-06T08:08:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:08:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-04-06T08:08:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:08:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:08:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:08:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:08:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-04-06T08:09:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-04-06T08:09:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_12': 2024-04-06T08:09:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12ea7c50239a 2024-04-06T08:09:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12ea7c50239a 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:59 2024 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:59 2024 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:59 2024 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8901ee92-51d2-43ee-985f-683952d0fb7e 2024-04-06T08:09:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_12 2024-04-06T08:09:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:09:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=8 2024-04-06T08:09:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 21624832 bytes 2024-04-06T08:09:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 95494144 bytes 2024-04-06T08:09:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 151684096 bytes 2024-04-06T08:09:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 199929856 bytes 2024-04-06T08:09:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 220770304 bytes 2024-04-06T08:09:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 249735168 bytes 2024-04-06T08:09:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 252706816 bytes 2024-04-06T08:09:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 264822784 bytes 2024-04-06T08:09:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 338488320 bytes 2024-04-06T08:09:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 378476544 bytes 2024-04-06T08:09:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 389452800 bytes 2024-04-06T08:09:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 452902912 bytes 2024-04-06T08:09:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 485479424 bytes 2024-04-06T08:09:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 552225792 bytes 2024-04-06T08:09:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 616895488 bytes 2024-04-06T08:09:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 638939136 bytes 2024-04-06T08:09:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 662293504 bytes 2024-04-06T08:09:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 735152128 bytes 2024-04-06T08:09:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 789758976 bytes 2024-04-06T08:09:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 824379392 bytes 2024-04-06T08:09:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 853598208 bytes 2024-04-06T08:09:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 890605568 bytes 2024-04-06T08:09:04.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1010016256 bytes 2024-04-06T08:09:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1063411712 bytes 2024-04-06T08:09:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1127198720 bytes 2024-04-06T08:09:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1128383488 bytes 2024-04-06T08:09:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1204885504 bytes 2024-04-06T08:09:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1319186432 bytes 2024-04-06T08:09:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1382475776 bytes 2024-04-06T08:09:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1484873728 bytes 2024-04-06T08:09:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1514874880 bytes 2024-04-06T08:09:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1538916352 bytes 2024-04-06T08:09:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1586918400 bytes 2024-04-06T08:09:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1640520704 bytes 2024-04-06T08:09:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1693523968 bytes 2024-04-06T08:09:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1759718400 bytes 2024-04-06T08:09:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1826661376 bytes 2024-04-06T08:09:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1867233280 bytes 2024-04-06T08:09:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1889587200 bytes 2024-04-06T08:09:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1890224128 bytes 2024-04-06T08:09:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1893277696 bytes 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_13 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-04-06T08:09:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_13 128M 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2024-04-06T08:09:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:09:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-04-06T08:09:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:09:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-04-06T08:09:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-04-06T08:09:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_13': 2024-04-06T08:09:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12f2bdc03649 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12f2bdc03649 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:08 2024 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:08 2024 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:08 2024 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 580a3f05-2cbb-49eb-bcd9-418388fdf451 2024-04-06T08:09:09.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_13 2024-04-06T08:09:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:09:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T08:09:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 614400 bytes 2024-04-06T08:09:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 32221184 bytes 2024-04-06T08:09:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 32825344 bytes 2024-04-06T08:09:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 90265600 bytes 2024-04-06T08:09:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 163905536 bytes 2024-04-06T08:09:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 191822848 bytes 2024-04-06T08:09:10.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 256557056 bytes 2024-04-06T08:09:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 327144448 bytes 2024-04-06T08:09:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 343929856 bytes 2024-04-06T08:09:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 394275840 bytes 2024-04-06T08:09:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 468264960 bytes 2024-04-06T08:09:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 503661568 bytes 2024-04-06T08:09:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 566567936 bytes 2024-04-06T08:09:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 606276608 bytes 2024-04-06T08:09:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 628137984 bytes 2024-04-06T08:09:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 672878592 bytes 2024-04-06T08:09:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 734191616 bytes 2024-04-06T08:09:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 782579712 bytes 2024-04-06T08:09:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 820020224 bytes 2024-04-06T08:09:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 835871744 bytes 2024-04-06T08:09:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 889530368 bytes 2024-04-06T08:09:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 922727424 bytes 2024-04-06T08:09:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 988352512 bytes 2024-04-06T08:09:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1032476672 bytes 2024-04-06T08:09:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1042119680 bytes 2024-04-06T08:09:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1083202560 bytes 2024-04-06T08:09:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1102555136 bytes 2024-04-06T08:09:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1179282432 bytes 2024-04-06T08:09:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1234362368 bytes 2024-04-06T08:09:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1287925760 bytes 2024-04-06T08:09:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1322824704 bytes 2024-04-06T08:09:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1439024128 bytes 2024-04-06T08:09:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1466643456 bytes 2024-04-06T08:09:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1483928576 bytes 2024-04-06T08:09:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1489016832 bytes 2024-04-06T08:09:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1549302784 bytes 2024-04-06T08:09:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1591361536 bytes 2024-04-06T08:09:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1614816256 bytes 2024-04-06T08:09:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_14 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_14 128M 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2024-04-06T08:09:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:09:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-04-06T08:09:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:09:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-04-06T08:09:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_14': 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12fbb53ebda 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12fbb53ebda 2024-04-06T08:09:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:16 2024 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:16 2024 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:16 2024 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: c583fe36-b4d5-459c-9629-9b6a5c24b1c2 2024-04-06T08:09:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_14 2024-04-06T08:09:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:09:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:09:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 50097152 bytes 2024-04-06T08:09:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 50715648 bytes 2024-04-06T08:09:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 140953600 bytes 2024-04-06T08:09:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 162996224 bytes 2024-04-06T08:09:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 187686912 bytes 2024-04-06T08:09:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 223379456 bytes 2024-04-06T08:09:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 233739264 bytes 2024-04-06T08:09:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 281698304 bytes 2024-04-06T08:09:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 282330112 bytes 2024-04-06T08:09:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 301093888 bytes 2024-04-06T08:09:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 362124288 bytes 2024-04-06T08:09:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 419554304 bytes 2024-04-06T08:09:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 432569344 bytes 2024-04-06T08:09:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 459711488 bytes 2024-04-06T08:09:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 491376640 bytes 2024-04-06T08:09:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 586681344 bytes 2024-04-06T08:09:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 638837760 bytes 2024-04-06T08:09:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 712010752 bytes 2024-04-06T08:09:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 740901888 bytes 2024-04-06T08:09:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 781188096 bytes 2024-04-06T08:09:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 795106304 bytes 2024-04-06T08:09:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 832676864 bytes 2024-04-06T08:09:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 843091968 bytes 2024-04-06T08:09:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 894995456 bytes 2024-04-06T08:09:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 895625216 bytes 2024-04-06T08:09:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 921481216 bytes 2024-04-06T08:09:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 935161856 bytes 2024-04-06T08:09:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 935788544 bytes 2024-04-06T08:09:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 936399872 bytes 2024-04-06T08:09:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 979444736 bytes 2024-04-06T08:09:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1011572736 bytes 2024-04-06T08:09:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1032003584 bytes 2024-04-06T08:09:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1097509888 bytes 2024-04-06T08:09:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1142374400 bytes 2024-04-06T08:09:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1179103232 bytes 2024-04-06T08:09:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1234588672 bytes 2024-04-06T08:09:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1261959168 bytes 2024-04-06T08:09:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1303482368 bytes 2024-04-06T08:09:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1342137344 bytes 2024-04-06T08:09:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1365913600 bytes 2024-04-06T08:09:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1423070208 bytes 2024-04-06T08:09:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1424742400 bytes 2024-04-06T08:09:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1465010176 bytes 2024-04-06T08:09:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1527770112 bytes 2024-04-06T08:09:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1584302080 bytes 2024-04-06T08:09:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1592249344 bytes 2024-04-06T08:09:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1621176320 bytes 2024-04-06T08:09:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1657962496 bytes 2024-04-06T08:09:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1665925120 bytes 2024-04-06T08:09:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1681333248 bytes 2024-04-06T08:09:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1741302784 bytes 2024-04-06T08:09:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1880721408 bytes 2024-04-06T08:09:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 1917735936 bytes 2024-04-06T08:09:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_15 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_15 128M 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:09:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2024-04-06T08:09:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-04-06T08:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-04-06T08:09:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_15': 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13057f43de1d 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13057f43de1d 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:26 2024 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:26 2024 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:26 2024 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 9ab9634b-f0c0-42f0-857e-b5d198938167 2024-04-06T08:09:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_15 2024-04-06T08:09:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:09:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T08:09:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 54609920 bytes 2024-04-06T08:09:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 147133440 bytes 2024-04-06T08:09:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 271645696 bytes 2024-04-06T08:09:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 298786816 bytes 2024-04-06T08:09:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 324021248 bytes 2024-04-06T08:09:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 402606080 bytes 2024-04-06T08:09:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 501228544 bytes 2024-04-06T08:09:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 616301568 bytes 2024-04-06T08:09:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 658766848 bytes 2024-04-06T08:09:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 666998784 bytes 2024-04-06T08:09:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 721770496 bytes 2024-04-06T08:09:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 744116224 bytes 2024-04-06T08:09:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 808764416 bytes 2024-04-06T08:09:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 859860992 bytes 2024-04-06T08:09:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 913531904 bytes 2024-04-06T08:09:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 923513856 bytes 2024-04-06T08:09:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 994274304 bytes 2024-04-06T08:09:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 1060496384 bytes 2024-04-06T08:09:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 1077528576 bytes 2024-04-06T08:09:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 1166954496 bytes 2024-04-06T08:09:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1207928832 bytes 2024-04-06T08:09:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1222308864 bytes 2024-04-06T08:09:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1222922240 bytes 2024-04-06T08:09:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1278187520 bytes 2024-04-06T08:09:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1348909056 bytes 2024-04-06T08:09:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1449803776 bytes 2024-04-06T08:09:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1475893248 bytes 2024-04-06T08:09:32.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1484377088 bytes 2024-04-06T08:09:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1537449984 bytes 2024-04-06T08:09:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1610532864 bytes 2024-04-06T08:09:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1646292992 bytes 2024-04-06T08:09:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1648660480 bytes 2024-04-06T08:09:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1674905600 bytes 2024-04-06T08:09:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1787558912 bytes 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_16 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_16 128M 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:09:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:09:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-04-06T08:09:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_16': 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 130eef15d362 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.130eef15d362 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:33 2024 2024-04-06T08:09:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:33 2024 2024-04-06T08:09:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:33 2024 2024-04-06T08:09:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 725496c1-43b8-4408-a32f-b66992d7c9ef 2024-04-06T08:09:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_16 2024-04-06T08:09:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:09:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=5 2024-04-06T08:09:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 43430912 bytes 2024-04-06T08:09:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 77024256 bytes 2024-04-06T08:09:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 129116160 bytes 2024-04-06T08:09:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 137513984 bytes 2024-04-06T08:09:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 161808384 bytes 2024-04-06T08:09:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 182202368 bytes 2024-04-06T08:09:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 234607616 bytes 2024-04-06T08:09:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 257917952 bytes 2024-04-06T08:09:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 268420096 bytes 2024-04-06T08:09:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 331957248 bytes 2024-04-06T08:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 338132992 bytes 2024-04-06T08:09:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 360464384 bytes 2024-04-06T08:09:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 406028288 bytes 2024-04-06T08:09:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 476039168 bytes 2024-04-06T08:09:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 510365696 bytes 2024-04-06T08:09:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 528219136 bytes 2024-04-06T08:09:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 571786240 bytes 2024-04-06T08:09:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 600445952 bytes 2024-04-06T08:09:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 672905216 bytes 2024-04-06T08:09:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 735192064 bytes 2024-04-06T08:09:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 803273728 bytes 2024-04-06T08:09:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 812199936 bytes 2024-04-06T08:09:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 825388032 bytes 2024-04-06T08:09:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 843350016 bytes 2024-04-06T08:09:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 927069184 bytes 2024-04-06T08:09:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 975185920 bytes 2024-04-06T08:09:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1012195328 bytes 2024-04-06T08:09:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1050417152 bytes 2024-04-06T08:09:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:09:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_17 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_17 128M 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2024-04-06T08:09:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:09:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-04-06T08:09:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:09:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-04-06T08:09:40.041 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4282938216050347) > weight (0.25) 2024-04-06T08:09:40.042 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6100722177746717) > weight (0.25) 2024-04-06T08:09:40.042 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5603416446935172) > weight (0.25) 2024-04-06T08:09:40.042 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9193241136141301) > weight (0.25) 2024-04-06T08:09:40.042 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-04-06T08:09:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-04-06T08:09:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_17': 2024-04-06T08:09:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13169169a04e 2024-04-06T08:09:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13169169a04e 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:39 2024 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:39 2024 2024-04-06T08:09:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:39 2024 2024-04-06T08:09:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ffce73d4-43fe-46e8-9362-ed0490dd6ab0 2024-04-06T08:09:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_17 2024-04-06T08:09:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:09:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:09:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 35454976 bytes 2024-04-06T08:09:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 119188480 bytes 2024-04-06T08:09:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 168871936 bytes 2024-04-06T08:09:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 224508928 bytes 2024-04-06T08:09:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 248430592 bytes 2024-04-06T08:09:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 296359936 bytes 2024-04-06T08:09:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 296978432 bytes 2024-04-06T08:09:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 299649024 bytes 2024-04-06T08:09:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 331034624 bytes 2024-04-06T08:09:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 383177728 bytes 2024-04-06T08:09:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 409673728 bytes 2024-04-06T08:09:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 482207744 bytes 2024-04-06T08:09:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 512017408 bytes 2024-04-06T08:09:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 624320512 bytes 2024-04-06T08:09:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 670935040 bytes 2024-04-06T08:09:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 749996032 bytes 2024-04-06T08:09:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 803737600 bytes 2024-04-06T08:09:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 824397824 bytes 2024-04-06T08:09:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 824984576 bytes 2024-04-06T08:09:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 848784384 bytes 2024-04-06T08:09:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 882361344 bytes 2024-04-06T08:09:44.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 895618048 bytes 2024-04-06T08:09:45.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 942807040 bytes 2024-04-06T08:09:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 967953408 bytes 2024-04-06T08:09:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1021166592 bytes 2024-04-06T08:09:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1036263424 bytes 2024-04-06T08:09:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1046334464 bytes 2024-04-06T08:09:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1068704768 bytes 2024-04-06T08:09:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1084043264 bytes 2024-04-06T08:09:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1106836480 bytes 2024-04-06T08:09:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1132734464 bytes 2024-04-06T08:09:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1164960768 bytes 2024-04-06T08:09:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1206168576 bytes 2024-04-06T08:09:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1254730752 bytes 2024-04-06T08:09:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1269821440 bytes 2024-04-06T08:09:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1291566080 bytes 2024-04-06T08:09:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1327775744 bytes 2024-04-06T08:09:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1389323264 bytes 2024-04-06T08:09:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1438644224 bytes 2024-04-06T08:09:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1455648768 bytes 2024-04-06T08:09:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1471620096 bytes 2024-04-06T08:09:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1526538240 bytes 2024-04-06T08:09:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1580432384 bytes 2024-04-06T08:09:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1643847680 bytes 2024-04-06T08:09:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1649367040 bytes 2024-04-06T08:09:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1649985536 bytes 2024-04-06T08:09:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1689541632 bytes 2024-04-06T08:09:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1713927168 bytes 2024-04-06T08:09:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1716669440 bytes 2024-04-06T08:09:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1823777792 bytes 2024-04-06T08:09:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1862498304 bytes 2024-04-06T08:09:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1893974016 bytes 2024-04-06T08:09:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2010028032 bytes 2024-04-06T08:09:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2030439424 bytes 2024-04-06T08:09:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2085883904 bytes 2024-04-06T08:09:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2145406976 bytes 2024-04-06T08:09:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2153772032 bytes 2024-04-06T08:09:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2259786752 bytes 2024-04-06T08:09:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2277143552 bytes 2024-04-06T08:09:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2292298752 bytes 2024-04-06T08:09:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2312478720 bytes 2024-04-06T08:09:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2329848832 bytes 2024-04-06T08:09:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2384442368 bytes 2024-04-06T08:09:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2418779136 bytes 2024-04-06T08:09:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2460798976 bytes 2024-04-06T08:09:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2504223744 bytes 2024-04-06T08:09:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2520820736 bytes 2024-04-06T08:09:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2590199808 bytes 2024-04-06T08:09:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2593794048 bytes 2024-04-06T08:09:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2626830336 bytes 2024-04-06T08:09:53.204 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:09:53.201+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d6702820 2024-04-06T08:09:53.204 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:09:53.201+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d6703a00 2024-04-06T08:09:53.204 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:09:53.201+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d90ac340 2024-04-06T08:09:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2643066880 bytes 2024-04-06T08:09:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2656972800 bytes 2024-04-06T08:09:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 2667480064 bytes 2024-04-06T08:09:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 2712262656 bytes 2024-04-06T08:09:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:09:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_18 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:09:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_18 128M 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:09:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:09:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-04-06T08:09:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:09:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-04-06T08:09:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_18': 2024-04-06T08:09:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:09:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:09:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:09:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1320716a086 2024-04-06T08:09:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1320716a086 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:53 2024 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:53 2024 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:53 2024 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 2a2e0364-70be-4a9d-8f86-d8dd2c9b39fc 2024-04-06T08:09:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:09:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:09:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_18 2024-04-06T08:09:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:09:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:09:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:09:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:09:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=8 2024-04-06T08:09:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:09:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:09:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 36502528 bytes 2024-04-06T08:09:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 78733312 bytes 2024-04-06T08:09:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 90872832 bytes 2024-04-06T08:09:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 134338560 bytes 2024-04-06T08:09:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 149521408 bytes 2024-04-06T08:09:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 173341696 bytes 2024-04-06T08:09:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 174227456 bytes 2024-04-06T08:09:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 216271872 bytes 2024-04-06T08:09:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 266193920 bytes 2024-04-06T08:09:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 346240000 bytes 2024-04-06T08:09:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 378118144 bytes 2024-04-06T08:09:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 434416640 bytes 2024-04-06T08:09:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 460450816 bytes 2024-04-06T08:09:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 507122688 bytes 2024-04-06T08:09:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 517944320 bytes 2024-04-06T08:09:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 572747776 bytes 2024-04-06T08:09:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 616289280 bytes 2024-04-06T08:09:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 652777472 bytes 2024-04-06T08:09:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 667169792 bytes 2024-04-06T08:09:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 677824512 bytes 2024-04-06T08:09:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 764069888 bytes 2024-04-06T08:09:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 837786624 bytes 2024-04-06T08:09:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 886526976 bytes 2024-04-06T08:09:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 887144448 bytes 2024-04-06T08:09:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 899921920 bytes 2024-04-06T08:09:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 942874624 bytes 2024-04-06T08:09:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1005328384 bytes 2024-04-06T08:09:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1022850048 bytes 2024-04-06T08:10:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1055292416 bytes 2024-04-06T08:10:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1084205056 bytes 2024-04-06T08:10:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1097861120 bytes 2024-04-06T08:10:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1108366336 bytes 2024-04-06T08:10:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1129669632 bytes 2024-04-06T08:10:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1182328832 bytes 2024-04-06T08:10:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1260685312 bytes 2024-04-06T08:10:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1307266048 bytes 2024-04-06T08:10:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1325935616 bytes 2024-04-06T08:10:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1339210752 bytes 2024-04-06T08:10:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1385836544 bytes 2024-04-06T08:10:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1422839808 bytes 2024-04-06T08:10:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1446615040 bytes 2024-04-06T08:10:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1472123904 bytes 2024-04-06T08:10:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1489841152 bytes 2024-04-06T08:10:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1545197568 bytes 2024-04-06T08:10:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1583491072 bytes 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_19 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-04-06T08:10:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_19 128M 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:10:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:10:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:10:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:10:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2024-04-06T08:10:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:10:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-04-06T08:10:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:10:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-04-06T08:10:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:10:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-04-06T08:10:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_19': 2024-04-06T08:10:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:10:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:10:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:10:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13285e7cdf19 2024-04-06T08:10:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13285e7cdf19 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:02 2024 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:02 2024 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:02 2024 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 73a422d7-74ad-4197-90fc-2e353882f67e 2024-04-06T08:10:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:10:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:10:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_19 2024-04-06T08:10:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:10:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:10:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:10:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:10:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:10:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 45024256 bytes 2024-04-06T08:10:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 59187200 bytes 2024-04-06T08:10:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 96568320 bytes 2024-04-06T08:10:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 115542016 bytes 2024-04-06T08:10:04.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 154677248 bytes 2024-04-06T08:10:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 160736256 bytes 2024-04-06T08:10:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 186728448 bytes 2024-04-06T08:10:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 201037824 bytes 2024-04-06T08:10:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 230525952 bytes 2024-04-06T08:10:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 238748672 bytes 2024-04-06T08:10:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 239346688 bytes 2024-04-06T08:10:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 324363264 bytes 2024-04-06T08:10:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 347346944 bytes 2024-04-06T08:10:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 370476032 bytes 2024-04-06T08:10:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 381813760 bytes 2024-04-06T08:10:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 426895360 bytes 2024-04-06T08:10:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 448601088 bytes 2024-04-06T08:10:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 508644352 bytes 2024-04-06T08:10:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 579474432 bytes 2024-04-06T08:10:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 580106240 bytes 2024-04-06T08:10:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 602177536 bytes 2024-04-06T08:10:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 624536576 bytes 2024-04-06T08:10:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 654831616 bytes 2024-04-06T08:10:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 695715840 bytes 2024-04-06T08:10:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 726297600 bytes 2024-04-06T08:10:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 739921920 bytes 2024-04-06T08:10:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 780198912 bytes 2024-04-06T08:10:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 795650048 bytes 2024-04-06T08:10:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 849120256 bytes 2024-04-06T08:10:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 930203648 bytes 2024-04-06T08:10:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1005597696 bytes 2024-04-06T08:10:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1051139072 bytes 2024-04-06T08:10:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1067806720 bytes 2024-04-06T08:10:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1088336896 bytes 2024-04-06T08:10:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1113961472 bytes 2024-04-06T08:10:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1157561344 bytes 2024-04-06T08:10:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1169264640 bytes 2024-04-06T08:10:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1216530432 bytes 2024-04-06T08:10:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1236516864 bytes 2024-04-06T08:10:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1293447168 bytes 2024-04-06T08:10:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1395044352 bytes 2024-04-06T08:10:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1395659776 bytes 2024-04-06T08:10:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1400069120 bytes 2024-04-06T08:10:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1436913664 bytes 2024-04-06T08:10:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1576061952 bytes 2024-04-06T08:10:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1635891200 bytes 2024-04-06T08:10:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1718865920 bytes 2024-04-06T08:10:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1777424384 bytes 2024-04-06T08:10:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1820522496 bytes 2024-04-06T08:10:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1863720960 bytes 2024-04-06T08:10:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1905811456 bytes 2024-04-06T08:10:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1940484096 bytes 2024-04-06T08:10:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2003368960 bytes 2024-04-06T08:10:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2039843840 bytes 2024-04-06T08:10:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2069309440 bytes 2024-04-06T08:10:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2153534464 bytes 2024-04-06T08:10:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2180303872 bytes 2024-04-06T08:10:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2258457600 bytes 2024-04-06T08:10:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2299774976 bytes 2024-04-06T08:10:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2386587648 bytes 2024-04-06T08:10:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2406118400 bytes 2024-04-06T08:10:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2434384896 bytes 2024-04-06T08:10:15.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2515916800 bytes 2024-04-06T08:10:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2534551552 bytes 2024-04-06T08:10:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2577607680 bytes 2024-04-06T08:10:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2628553728 bytes 2024-04-06T08:10:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:10:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:10:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:10:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:10:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:10:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_20 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_20 128M 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-04-06T08:10:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:10:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-04-06T08:10:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_20': 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1332c57f9f13 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1332c57f9f13 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:15 2024 2024-04-06T08:10:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:15 2024 2024-04-06T08:10:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:15 2024 2024-04-06T08:10:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:10:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:10:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 877e6897-26c6-469a-b566-eb8cf289549d 2024-04-06T08:10:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:10:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:10:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_20 2024-04-06T08:10:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:10:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:10:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=5 2024-04-06T08:10:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:10:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:10:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 45797376 bytes 2024-04-06T08:10:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 112677888 bytes 2024-04-06T08:10:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 115994624 bytes 2024-04-06T08:10:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 163295232 bytes 2024-04-06T08:10:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 217252864 bytes 2024-04-06T08:10:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 254973952 bytes 2024-04-06T08:10:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 281033728 bytes 2024-04-06T08:10:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 334919680 bytes 2024-04-06T08:10:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 409052160 bytes 2024-04-06T08:10:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 456380416 bytes 2024-04-06T08:10:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 466212864 bytes 2024-04-06T08:10:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 526557184 bytes 2024-04-06T08:10:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 571157504 bytes 2024-04-06T08:10:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 635668480 bytes 2024-04-06T08:10:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 672962560 bytes 2024-04-06T08:10:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 702284800 bytes 2024-04-06T08:10:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 702900224 bytes 2024-04-06T08:10:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 712475648 bytes 2024-04-06T08:10:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 789697536 bytes 2024-04-06T08:10:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 875497472 bytes 2024-04-06T08:10:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 885260288 bytes 2024-04-06T08:10:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 911935488 bytes 2024-04-06T08:10:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 921024512 bytes 2024-04-06T08:10:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 925011968 bytes 2024-04-06T08:10:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 957481984 bytes 2024-04-06T08:10:20.943 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T08:10:20.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T08:10:20.944 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-06T08:10:20.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:10:20.946+0000 7f0f4ae58640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 35668) UID: 1000 2024-04-06T08:10:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 993510400 bytes 2024-04-06T08:10:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1023082496 bytes 2024-04-06T08:10:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1100708864 bytes 2024-04-06T08:10:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:10:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:10:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:10:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:10:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_21 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_21 128M 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2024-04-06T08:10:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:10:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-04-06T08:10:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:10:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-04-06T08:10:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:10:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-04-06T08:10:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_21': 2024-04-06T08:10:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:10:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:10:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:10:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 133c6cd58ac3 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.133c6cd58ac3 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:21 2024 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:21 2024 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:21 2024 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: a30e9377-aa3c-4d27-9702-2aab3727fd99 2024-04-06T08:10:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:10:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:10:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_21 2024-04-06T08:10:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:10:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:10:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T08:10:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:10:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:10:21.944 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T08:10:21.944 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:10:21.945 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T08:10:21.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T08:10:21.945 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T08:10:21.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T08:10:21.949 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-04-06T08:10:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 27101184 bytes 2024-04-06T08:10:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 55977984 bytes 2024-04-06T08:10:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 109760512 bytes 2024-04-06T08:10:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 154958848 bytes 2024-04-06T08:10:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 156702720 bytes 2024-04-06T08:10:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 211439616 bytes 2024-04-06T08:10:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 238279680 bytes 2024-04-06T08:10:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 285489152 bytes 2024-04-06T08:10:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 310539264 bytes 2024-04-06T08:10:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 340889600 bytes 2024-04-06T08:10:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 415825920 bytes 2024-04-06T08:10:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 443167744 bytes 2024-04-06T08:10:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 512893952 bytes 2024-04-06T08:10:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 550617088 bytes 2024-04-06T08:10:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 594898944 bytes 2024-04-06T08:10:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 659126272 bytes 2024-04-06T08:10:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 669565952 bytes 2024-04-06T08:10:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 707529728 bytes 2024-04-06T08:10:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 716893184 bytes 2024-04-06T08:10:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 776272896 bytes 2024-04-06T08:10:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 827161600 bytes 2024-04-06T08:10:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 874875904 bytes 2024-04-06T08:10:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 956878848 bytes 2024-04-06T08:10:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1038365696 bytes 2024-04-06T08:10:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1098366976 bytes 2024-04-06T08:10:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1155325952 bytes 2024-04-06T08:10:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1195386880 bytes 2024-04-06T08:10:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1216837632 bytes 2024-04-06T08:10:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1272081408 bytes 2024-04-06T08:10:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1289064448 bytes 2024-04-06T08:10:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1318252544 bytes 2024-04-06T08:10:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1358821376 bytes 2024-04-06T08:10:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1406651392 bytes 2024-04-06T08:10:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1407240192 bytes 2024-04-06T08:10:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1444326400 bytes 2024-04-06T08:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1459791872 bytes 2024-04-06T08:10:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1512483840 bytes 2024-04-06T08:10:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1566803968 bytes 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_22 2024-04-06T08:10:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_22 128M 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:10:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:10:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:10:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:10:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-04-06T08:10:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:10:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_22': 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1349ecd2ea7e 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1349ecd2ea7e 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:28 2024 2024-04-06T08:10:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:28 2024 2024-04-06T08:10:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:28 2024 2024-04-06T08:10:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:10:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:10:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ac3a5da1-f39e-465e-a835-c87e8aa02575 2024-04-06T08:10:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:10:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:10:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_22 2024-04-06T08:10:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:10:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:10:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:10:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:10:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:10:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 38766592 bytes 2024-04-06T08:10:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 79251456 bytes 2024-04-06T08:10:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 94824448 bytes 2024-04-06T08:10:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 117950464 bytes 2024-04-06T08:10:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 150322176 bytes 2024-04-06T08:10:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 212080640 bytes 2024-04-06T08:10:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 233871360 bytes 2024-04-06T08:10:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 316216320 bytes 2024-04-06T08:10:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 362592256 bytes 2024-04-06T08:10:31.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 441009152 bytes 2024-04-06T08:10:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 462498816 bytes 2024-04-06T08:10:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 531560448 bytes 2024-04-06T08:10:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 557780992 bytes 2024-04-06T08:10:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 574726144 bytes 2024-04-06T08:10:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 626019328 bytes 2024-04-06T08:10:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 692587520 bytes 2024-04-06T08:10:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 754160640 bytes 2024-04-06T08:10:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 789931008 bytes 2024-04-06T08:10:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 790544384 bytes 2024-04-06T08:10:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 791141376 bytes 2024-04-06T08:10:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 848389120 bytes 2024-04-06T08:10:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 907313152 bytes 2024-04-06T08:10:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 930653184 bytes 2024-04-06T08:10:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1019242496 bytes 2024-04-06T08:10:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1034019840 bytes 2024-04-06T08:10:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1091708928 bytes 2024-04-06T08:10:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1158745088 bytes 2024-04-06T08:10:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1314329600 bytes 2024-04-06T08:10:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1348974592 bytes 2024-04-06T08:10:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1361557504 bytes 2024-04-06T08:10:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1417160704 bytes 2024-04-06T08:10:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1433031680 bytes 2024-04-06T08:10:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1501220864 bytes 2024-04-06T08:10:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1568928768 bytes 2024-04-06T08:10:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1620039680 bytes 2024-04-06T08:10:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1692431360 bytes 2024-04-06T08:10:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1744513024 bytes 2024-04-06T08:10:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1823799296 bytes 2024-04-06T08:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1850209280 bytes 2024-04-06T08:10:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1865126912 bytes 2024-04-06T08:10:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1909482496 bytes 2024-04-06T08:10:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1925166080 bytes 2024-04-06T08:10:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1994107904 bytes 2024-04-06T08:10:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2034338816 bytes 2024-04-06T08:10:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2108999680 bytes 2024-04-06T08:10:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2159821824 bytes 2024-04-06T08:10:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2199010304 bytes 2024-04-06T08:10:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2216254464 bytes 2024-04-06T08:10:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2269087744 bytes 2024-04-06T08:10:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2326178816 bytes 2024-04-06T08:10:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2401435648 bytes 2024-04-06T08:10:40.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8098211606488299) > weight (0.25) 2024-04-06T08:10:40.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8210936966716832) > weight (0.25) 2024-04-06T08:10:40.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.47113820448408916) > weight (0.25) 2024-04-06T08:10:40.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3203539602195403) > weight (0.25) 2024-04-06T08:10:40.043 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-04-06T08:10:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2417905664 bytes 2024-04-06T08:10:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2471605248 bytes 2024-04-06T08:10:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2517387264 bytes 2024-04-06T08:10:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2592500736 bytes 2024-04-06T08:10:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2634614784 bytes 2024-04-06T08:10:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2659303424 bytes 2024-04-06T08:10:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2691098624 bytes 2024-04-06T08:10:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2705881088 bytes 2024-04-06T08:10:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2722998272 bytes 2024-04-06T08:10:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2816710656 bytes 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_23 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_23 128M 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:10:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:10:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2024-04-06T08:10:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:10:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-04-06T08:10:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:10:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-04-06T08:10:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:10:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_23': 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 135251d87a28 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.135251d87a28 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:41 2024 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:41 2024 2024-04-06T08:10:42.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:41 2024 2024-04-06T08:10:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:10:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:10:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ac6d51f9-12c0-4020-8ba6-c41cc112c761 2024-04-06T08:10:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:10:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:10:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_23 2024-04-06T08:10:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:10:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:10:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=11 2024-04-06T08:10:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:10:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:10:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 52965376 bytes 2024-04-06T08:10:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 117405696 bytes 2024-04-06T08:10:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 175409152 bytes 2024-04-06T08:10:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 202515456 bytes 2024-04-06T08:10:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 248143872 bytes 2024-04-06T08:10:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 281219072 bytes 2024-04-06T08:10:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 300097536 bytes 2024-04-06T08:10:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 331411456 bytes 2024-04-06T08:10:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 332765184 bytes 2024-04-06T08:10:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 426271744 bytes 2024-04-06T08:10:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 509449216 bytes 2024-04-06T08:10:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 536819712 bytes 2024-04-06T08:10:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 619506688 bytes 2024-04-06T08:10:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 709412864 bytes 2024-04-06T08:10:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 710049792 bytes 2024-04-06T08:10:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 771035136 bytes 2024-04-06T08:10:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 806723584 bytes 2024-04-06T08:10:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 874000384 bytes 2024-04-06T08:10:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 946318336 bytes 2024-04-06T08:10:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 966549504 bytes 2024-04-06T08:10:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 982183936 bytes 2024-04-06T08:10:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1001083904 bytes 2024-04-06T08:10:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1068956672 bytes 2024-04-06T08:10:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1069576192 bytes 2024-04-06T08:10:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1121461248 bytes 2024-04-06T08:10:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1162432512 bytes 2024-04-06T08:10:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1196854272 bytes 2024-04-06T08:10:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1215847424 bytes 2024-04-06T08:10:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1250449408 bytes 2024-04-06T08:10:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1284371456 bytes 2024-04-06T08:10:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1308986368 bytes 2024-04-06T08:10:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1421744128 bytes 2024-04-06T08:10:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1462326272 bytes 2024-04-06T08:10:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1489491968 bytes 2024-04-06T08:10:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1569021952 bytes 2024-04-06T08:10:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1635070976 bytes 2024-04-06T08:10:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1664735232 bytes 2024-04-06T08:10:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1725976576 bytes 2024-04-06T08:10:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1762897920 bytes 2024-04-06T08:10:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1814682624 bytes 2024-04-06T08:10:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1832878080 bytes 2024-04-06T08:10:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1843131392 bytes 2024-04-06T08:10:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1862946816 bytes 2024-04-06T08:10:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1905131520 bytes 2024-04-06T08:10:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1914093568 bytes 2024-04-06T08:10:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1952599040 bytes 2024-04-06T08:10:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1989755904 bytes 2024-04-06T08:10:52.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2028072960 bytes 2024-04-06T08:10:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2089711616 bytes 2024-04-06T08:10:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2135884800 bytes 2024-04-06T08:10:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2148849664 bytes 2024-04-06T08:10:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2197354496 bytes 2024-04-06T08:10:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2307294208 bytes 2024-04-06T08:10:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2307911680 bytes 2024-04-06T08:10:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2355231744 bytes 2024-04-06T08:10:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2417170432 bytes 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_24 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_24 128M 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:10:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:10:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:10:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:10:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:10:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2024-04-06T08:10:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:10:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-04-06T08:10:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:10:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:10:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-04-06T08:10:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:10:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-04-06T08:10:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_24': 2024-04-06T08:10:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:10:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:10:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:10:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 135a7e9e44b1 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.135a7e9e44b1 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:53 2024 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:53 2024 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:53 2024 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8f4562b6-b8d0-438b-a48f-defcceca733b 2024-04-06T08:10:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:10:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:10:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_24 2024-04-06T08:10:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:10:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:10:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:10:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:10:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:10:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:10:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:10:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 87411712 bytes 2024-04-06T08:10:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 144035840 bytes 2024-04-06T08:10:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 203208704 bytes 2024-04-06T08:10:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 237670400 bytes 2024-04-06T08:10:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 305819648 bytes 2024-04-06T08:10:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 414034944 bytes 2024-04-06T08:10:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 471215104 bytes 2024-04-06T08:10:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 506856448 bytes 2024-04-06T08:10:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 519953408 bytes 2024-04-06T08:10:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 572081152 bytes 2024-04-06T08:10:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 626418688 bytes 2024-04-06T08:10:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 653974528 bytes 2024-04-06T08:10:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 694089728 bytes 2024-04-06T08:10:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 738802688 bytes 2024-04-06T08:10:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 757165056 bytes 2024-04-06T08:10:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 759715840 bytes 2024-04-06T08:10:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 852000768 bytes 2024-04-06T08:10:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 891157504 bytes 2024-04-06T08:10:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 960625664 bytes 2024-04-06T08:10:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 1027757056 bytes 2024-04-06T08:10:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1034409984 bytes 2024-04-06T08:10:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1054694400 bytes 2024-04-06T08:10:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1079546880 bytes 2024-04-06T08:10:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1104685056 bytes 2024-04-06T08:10:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1202138112 bytes 2024-04-06T08:11:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1240938496 bytes 2024-04-06T08:11:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1273318400 bytes 2024-04-06T08:11:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1343005696 bytes 2024-04-06T08:11:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1364769792 bytes 2024-04-06T08:11:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1387698176 bytes 2024-04-06T08:11:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1406461952 bytes 2024-04-06T08:11:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1433175040 bytes 2024-04-06T08:11:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1455779840 bytes 2024-04-06T08:11:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1467612160 bytes 2024-04-06T08:11:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1472322560 bytes 2024-04-06T08:11:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1545949184 bytes 2024-04-06T08:11:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1589765120 bytes 2024-04-06T08:11:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1698366464 bytes 2024-04-06T08:11:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1706302464 bytes 2024-04-06T08:11:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1765741568 bytes 2024-04-06T08:11:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1822209024 bytes 2024-04-06T08:11:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1864147968 bytes 2024-04-06T08:11:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1956226048 bytes 2024-04-06T08:11:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1994748928 bytes 2024-04-06T08:11:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2001796096 bytes 2024-04-06T08:11:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2043750400 bytes 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_25 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:11:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_25 128M 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:11:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2024-04-06T08:11:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:11:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-04-06T08:11:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:11:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-04-06T08:11:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:11:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-04-06T08:11:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_25': 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1363a9d17abe 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1363a9d17abe 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:03 2024 2024-04-06T08:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:03 2024 2024-04-06T08:11:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:03 2024 2024-04-06T08:11:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:11:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:11:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f2c3f78a-8f35-4a3c-b4ba-4152fd15fcbe 2024-04-06T08:11:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:11:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:11:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_25 2024-04-06T08:11:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:11:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:11:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:11:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:11:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:11:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 23369728 bytes 2024-04-06T08:11:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 83883008 bytes 2024-04-06T08:11:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 175813632 bytes 2024-04-06T08:11:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 212284416 bytes 2024-04-06T08:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 294841344 bytes 2024-04-06T08:11:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 314774528 bytes 2024-04-06T08:11:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 333703168 bytes 2024-04-06T08:11:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 352603136 bytes 2024-04-06T08:11:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 397384704 bytes 2024-04-06T08:11:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 419141632 bytes 2024-04-06T08:11:06.860 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T08:11:06.860+0000 7fc4119a2640 -1 reset not still connected to 0x560b342f65b0 2024-04-06T08:11:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 477404160 bytes 2024-04-06T08:11:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 561508352 bytes 2024-04-06T08:11:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 577957888 bytes 2024-04-06T08:11:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 613055488 bytes 2024-04-06T08:11:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 647658496 bytes 2024-04-06T08:11:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 700537856 bytes 2024-04-06T08:11:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 761860096 bytes 2024-04-06T08:11:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 771792896 bytes 2024-04-06T08:11:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 805302272 bytes 2024-04-06T08:11:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 853689344 bytes 2024-04-06T08:11:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 934864896 bytes 2024-04-06T08:11:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 940343296 bytes 2024-04-06T08:11:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 963193856 bytes 2024-04-06T08:11:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1016040448 bytes 2024-04-06T08:11:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1059553280 bytes 2024-04-06T08:11:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1096773632 bytes 2024-04-06T08:11:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1128491008 bytes 2024-04-06T08:11:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1178571776 bytes 2024-04-06T08:11:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1217146880 bytes 2024-04-06T08:11:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1258375168 bytes 2024-04-06T08:11:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1321781248 bytes 2024-04-06T08:11:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1322416128 bytes 2024-04-06T08:11:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1353651200 bytes 2024-04-06T08:11:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1402023936 bytes 2024-04-06T08:11:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1441623040 bytes 2024-04-06T08:11:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1448619008 bytes 2024-04-06T08:11:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1577082880 bytes 2024-04-06T08:11:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1626566656 bytes 2024-04-06T08:11:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1680116736 bytes 2024-04-06T08:11:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1690035200 bytes 2024-04-06T08:11:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1764831232 bytes 2024-04-06T08:11:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1808788480 bytes 2024-04-06T08:11:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1866592256 bytes 2024-04-06T08:11:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1952223232 bytes 2024-04-06T08:11:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1987786752 bytes 2024-04-06T08:11:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1999556608 bytes 2024-04-06T08:11:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2068105216 bytes 2024-04-06T08:11:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2080685056 bytes 2024-04-06T08:11:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2081314816 bytes 2024-04-06T08:11:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2200286208 bytes 2024-04-06T08:11:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2200895488 bytes 2024-04-06T08:11:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2265605120 bytes 2024-04-06T08:11:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2274310144 bytes 2024-04-06T08:11:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2312496128 bytes 2024-04-06T08:11:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2350062592 bytes 2024-04-06T08:11:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2366611456 bytes 2024-04-06T08:11:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2442171392 bytes 2024-04-06T08:11:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2467268608 bytes 2024-04-06T08:11:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2511545344 bytes 2024-04-06T08:11:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2512421888 bytes 2024-04-06T08:11:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2541483008 bytes 2024-04-06T08:11:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2556593152 bytes 2024-04-06T08:11:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2559997952 bytes 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_26 2024-04-06T08:11:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_26 128M 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-04-06T08:11:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:11:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-04-06T08:11:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_26': 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 136d933d40d2 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.136d933d40d2 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:16 2024 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:16 2024 2024-04-06T08:11:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:16 2024 2024-04-06T08:11:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:11:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:11:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: e1986487-bd67-45b7-a50c-21436fe14d4d 2024-04-06T08:11:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:11:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:11:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_26 2024-04-06T08:11:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:11:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:11:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=10 2024-04-06T08:11:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:11:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:11:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 26669056 bytes 2024-04-06T08:11:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 67572736 bytes 2024-04-06T08:11:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 111811584 bytes 2024-04-06T08:11:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 129339392 bytes 2024-04-06T08:11:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 184370176 bytes 2024-04-06T08:11:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 188534784 bytes 2024-04-06T08:11:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 209250304 bytes 2024-04-06T08:11:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 275842048 bytes 2024-04-06T08:11:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 308431872 bytes 2024-04-06T08:11:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 329119744 bytes 2024-04-06T08:11:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 354114560 bytes 2024-04-06T08:11:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 365416448 bytes 2024-04-06T08:11:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 395311104 bytes 2024-04-06T08:11:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 426641408 bytes 2024-04-06T08:11:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 461346816 bytes 2024-04-06T08:11:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 537834496 bytes 2024-04-06T08:11:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 591648768 bytes 2024-04-06T08:11:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 661006336 bytes 2024-04-06T08:11:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 664306688 bytes 2024-04-06T08:11:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 724411392 bytes 2024-04-06T08:11:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 798673920 bytes 2024-04-06T08:11:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 824035328 bytes 2024-04-06T08:11:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 864581632 bytes 2024-04-06T08:11:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 947346432 bytes 2024-04-06T08:11:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1007468544 bytes 2024-04-06T08:11:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1044256768 bytes 2024-04-06T08:11:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1093006336 bytes 2024-04-06T08:11:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1148591104 bytes 2024-04-06T08:11:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1206547456 bytes 2024-04-06T08:11:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1218919424 bytes 2024-04-06T08:11:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1242933248 bytes 2024-04-06T08:11:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1295547392 bytes 2024-04-06T08:11:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1336862720 bytes 2024-04-06T08:11:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1354067968 bytes 2024-04-06T08:11:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1396717568 bytes 2024-04-06T08:11:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1500524544 bytes 2024-04-06T08:11:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1550701568 bytes 2024-04-06T08:11:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1575469056 bytes 2024-04-06T08:11:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1606672384 bytes 2024-04-06T08:11:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1652067328 bytes 2024-04-06T08:11:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1720700928 bytes 2024-04-06T08:11:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1843267584 bytes 2024-04-06T08:11:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1888021504 bytes 2024-04-06T08:11:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1935874048 bytes 2024-04-06T08:11:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1957093376 bytes 2024-04-06T08:11:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2047489024 bytes 2024-04-06T08:11:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2071883776 bytes 2024-04-06T08:11:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2072513536 bytes 2024-04-06T08:11:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2110912512 bytes 2024-04-06T08:11:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2143790080 bytes 2024-04-06T08:11:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2211811328 bytes 2024-04-06T08:11:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2231520256 bytes 2024-04-06T08:11:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2261560320 bytes 2024-04-06T08:11:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:11:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:11:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:11:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_27 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_27 128M 2024-04-06T08:11:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-04-06T08:11:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:11:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-04-06T08:11:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_27': 2024-04-06T08:11:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 137544177cbd 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.137544177cbd 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:27 2024 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:27 2024 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:27 2024 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:11:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:11:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8b927270-e88c-47d8-9f3e-375b09dbf2b2 2024-04-06T08:11:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:11:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:11:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_27 2024-04-06T08:11:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:11:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:11:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:11:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:11:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:11:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 46849024 bytes 2024-04-06T08:11:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 99159040 bytes 2024-04-06T08:11:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 200980480 bytes 2024-04-06T08:11:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 246313984 bytes 2024-04-06T08:11:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 319356928 bytes 2024-04-06T08:11:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 408992768 bytes 2024-04-06T08:11:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 472384512 bytes 2024-04-06T08:11:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 585231360 bytes 2024-04-06T08:11:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 632707072 bytes 2024-04-06T08:11:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 709005312 bytes 2024-04-06T08:11:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 762420224 bytes 2024-04-06T08:11:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 796761088 bytes 2024-04-06T08:11:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 841223168 bytes 2024-04-06T08:11:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 911505408 bytes 2024-04-06T08:11:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 981044224 bytes 2024-04-06T08:11:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 990633984 bytes 2024-04-06T08:11:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 1028263936 bytes 2024-04-06T08:11:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 1080614912 bytes 2024-04-06T08:11:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 1112379392 bytes 2024-04-06T08:11:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 1122260992 bytes 2024-04-06T08:11:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1122877440 bytes 2024-04-06T08:11:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1199100928 bytes 2024-04-06T08:11:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1234868224 bytes 2024-04-06T08:11:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1286327296 bytes 2024-04-06T08:11:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1353229312 bytes 2024-04-06T08:11:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1360314368 bytes 2024-04-06T08:11:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1400290304 bytes 2024-04-06T08:11:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1503383552 bytes 2024-04-06T08:11:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1550992384 bytes 2024-04-06T08:11:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1561996288 bytes 2024-04-06T08:11:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1637956608 bytes 2024-04-06T08:11:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1676728320 bytes 2024-04-06T08:11:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1706827776 bytes 2024-04-06T08:11:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1735004160 bytes 2024-04-06T08:11:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1762579456 bytes 2024-04-06T08:11:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1844458496 bytes 2024-04-06T08:11:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1927952384 bytes 2024-04-06T08:11:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1954476032 bytes 2024-04-06T08:11:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1983376384 bytes 2024-04-06T08:11:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 2067963904 bytes 2024-04-06T08:11:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 2103079936 bytes 2024-04-06T08:11:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 2129112064 bytes 2024-04-06T08:11:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2179014656 bytes 2024-04-06T08:11:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2242672640 bytes 2024-04-06T08:11:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2286808064 bytes 2024-04-06T08:11:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2307070976 bytes 2024-04-06T08:11:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2342935552 bytes 2024-04-06T08:11:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2412231680 bytes 2024-04-06T08:11:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2459988992 bytes 2024-04-06T08:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2545327104 bytes 2024-04-06T08:11:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2572040192 bytes 2024-04-06T08:11:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2586903552 bytes 2024-04-06T08:11:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2615309312 bytes 2024-04-06T08:11:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2654185472 bytes 2024-04-06T08:11:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2727825408 bytes 2024-04-06T08:11:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2797449216 bytes 2024-04-06T08:11:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2819394560 bytes 2024-04-06T08:11:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2892162048 bytes 2024-04-06T08:11:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2949880832 bytes 2024-04-06T08:11:40.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2950985728 bytes 2024-04-06T08:11:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2966350848 bytes 2024-04-06T08:11:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2978476032 bytes 2024-04-06T08:11:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2979091456 bytes 2024-04-06T08:11:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:11:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:11:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:11:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_28 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_28 128M 2024-04-06T08:11:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2024-04-06T08:11:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:11:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-04-06T08:11:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:11:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-04-06T08:11:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:11:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_28': 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 137e749c3a48 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.137e749c3a48 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:11:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:40 2024 2024-04-06T08:11:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:40 2024 2024-04-06T08:11:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:40 2024 2024-04-06T08:11:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:11:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:11:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 3c43d300-9de9-4d2a-ba10-0d577a7a18c0 2024-04-06T08:11:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:11:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:11:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_28 2024-04-06T08:11:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:11:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:11:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=11 2024-04-06T08:11:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:11:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:11:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 16068608 bytes 2024-04-06T08:11:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 56024064 bytes 2024-04-06T08:11:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 93853696 bytes 2024-04-06T08:11:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 105906176 bytes 2024-04-06T08:11:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 148218880 bytes 2024-04-06T08:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 170951680 bytes 2024-04-06T08:11:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 227312640 bytes 2024-04-06T08:11:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 285801472 bytes 2024-04-06T08:11:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 310754304 bytes 2024-04-06T08:11:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 378829824 bytes 2024-04-06T08:11:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 408436736 bytes 2024-04-06T08:11:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 451510272 bytes 2024-04-06T08:11:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 494021632 bytes 2024-04-06T08:11:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 563903488 bytes 2024-04-06T08:11:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 600856576 bytes 2024-04-06T08:11:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 677649408 bytes 2024-04-06T08:11:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 705909760 bytes 2024-04-06T08:11:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 745218048 bytes 2024-04-06T08:11:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 848910336 bytes 2024-04-06T08:11:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 866061312 bytes 2024-04-06T08:11:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 889015296 bytes 2024-04-06T08:11:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 914890752 bytes 2024-04-06T08:11:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 962152448 bytes 2024-04-06T08:11:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1040228352 bytes 2024-04-06T08:11:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1102726144 bytes 2024-04-06T08:11:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1103351808 bytes 2024-04-06T08:11:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1104380928 bytes 2024-04-06T08:11:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1160921088 bytes 2024-04-06T08:11:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1202028544 bytes 2024-04-06T08:11:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1229971456 bytes 2024-04-06T08:11:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1231765504 bytes 2024-04-06T08:11:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1277890560 bytes 2024-04-06T08:11:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1300420608 bytes 2024-04-06T08:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1349680128 bytes 2024-04-06T08:11:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1369059328 bytes 2024-04-06T08:11:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1391887360 bytes 2024-04-06T08:11:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1464645632 bytes 2024-04-06T08:11:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1550146560 bytes 2024-04-06T08:11:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1569306624 bytes 2024-04-06T08:11:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1569926144 bytes 2024-04-06T08:11:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1606571008 bytes 2024-04-06T08:11:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1610187776 bytes 2024-04-06T08:11:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1671553024 bytes 2024-04-06T08:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1742453760 bytes 2024-04-06T08:11:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1743040512 bytes 2024-04-06T08:11:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1805109248 bytes 2024-04-06T08:11:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1884207104 bytes 2024-04-06T08:11:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1962776576 bytes 2024-04-06T08:11:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1997524992 bytes 2024-04-06T08:11:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2025934848 bytes 2024-04-06T08:11:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2047746048 bytes 2024-04-06T08:11:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2120082432 bytes 2024-04-06T08:11:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2133380096 bytes 2024-04-06T08:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2185018368 bytes 2024-04-06T08:11:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2257903616 bytes 2024-04-06T08:11:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2279584768 bytes 2024-04-06T08:11:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2337777664 bytes 2024-04-06T08:11:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2377982976 bytes 2024-04-06T08:11:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:11:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:11:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:11:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:11:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_29 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_29 128M 2024-04-06T08:11:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2024-04-06T08:11:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:11:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-04-06T08:11:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:11:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-04-06T08:11:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:11:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-04-06T08:11:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_29': 2024-04-06T08:11:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:11:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:11:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:11:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1386f607156d 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1386f607156d 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:52 2024 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:52 2024 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:52 2024 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: c5b8cfe3-a440-4aa6-b7ed-bbf8e31b2175 2024-04-06T08:11:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:11:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:11:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_29 2024-04-06T08:11:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:11:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:11:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=5 2024-04-06T08:11:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:11:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:11:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 47690752 bytes 2024-04-06T08:11:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 84826112 bytes 2024-04-06T08:11:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 111599616 bytes 2024-04-06T08:11:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 141588480 bytes 2024-04-06T08:11:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 188966912 bytes 2024-04-06T08:11:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 236687360 bytes 2024-04-06T08:11:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 308130816 bytes 2024-04-06T08:11:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 413789184 bytes 2024-04-06T08:11:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 463965184 bytes 2024-04-06T08:11:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 518969344 bytes 2024-04-06T08:11:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 580555776 bytes 2024-04-06T08:11:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 600836096 bytes 2024-04-06T08:11:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 628240384 bytes 2024-04-06T08:11:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 647179264 bytes 2024-04-06T08:11:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 672538624 bytes 2024-04-06T08:11:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 712260608 bytes 2024-04-06T08:11:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 762605568 bytes 2024-04-06T08:11:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 796872704 bytes 2024-04-06T08:11:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 865360896 bytes 2024-04-06T08:11:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 973590528 bytes 2024-04-06T08:11:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1012070400 bytes 2024-04-06T08:11:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1082491904 bytes 2024-04-06T08:11:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1128286208 bytes 2024-04-06T08:11:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1204888576 bytes 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_30 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-04-06T08:11:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_30 128M 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:11:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2024-04-06T08:11:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:11:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-04-06T08:11:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:11:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:11:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-04-06T08:11:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:11:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-04-06T08:11:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_30': 2024-04-06T08:11:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:11:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:11:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1390e33ecb78 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1390e33ecb78 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:58 2024 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:58 2024 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:58 2024 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 0ac3adc2-3f86-4a07-a54a-4fd87c334a97 2024-04-06T08:11:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:11:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:11:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_30 2024-04-06T08:11:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:11:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:11:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:11:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:11:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:11:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:11:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:12:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 17561600 bytes 2024-04-06T08:12:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 52692992 bytes 2024-04-06T08:12:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 71200768 bytes 2024-04-06T08:12:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 71835648 bytes 2024-04-06T08:12:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 175910912 bytes 2024-04-06T08:12:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 190466048 bytes 2024-04-06T08:12:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 210525184 bytes 2024-04-06T08:12:01.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 223439872 bytes 2024-04-06T08:12:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 262237184 bytes 2024-04-06T08:12:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 262859776 bytes 2024-04-06T08:12:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 302869504 bytes 2024-04-06T08:12:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 417626112 bytes 2024-04-06T08:12:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 484556800 bytes 2024-04-06T08:12:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 521751552 bytes 2024-04-06T08:12:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 522366976 bytes 2024-04-06T08:12:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 573582336 bytes 2024-04-06T08:12:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 648135680 bytes 2024-04-06T08:12:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 671734784 bytes 2024-04-06T08:12:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 733516800 bytes 2024-04-06T08:12:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 754794496 bytes 2024-04-06T08:12:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 799333376 bytes 2024-04-06T08:12:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 828910592 bytes 2024-04-06T08:12:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 844035072 bytes 2024-04-06T08:12:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 883719168 bytes 2024-04-06T08:12:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 939045888 bytes 2024-04-06T08:12:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 939890688 bytes 2024-04-06T08:12:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 999958528 bytes 2024-04-06T08:12:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1002328064 bytes 2024-04-06T08:12:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1065830400 bytes 2024-04-06T08:12:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1117743104 bytes 2024-04-06T08:12:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1165350912 bytes 2024-04-06T08:12:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1193580544 bytes 2024-04-06T08:12:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1274729472 bytes 2024-04-06T08:12:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1299918848 bytes 2024-04-06T08:12:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1368627200 bytes 2024-04-06T08:12:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1424239616 bytes 2024-04-06T08:12:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1483714560 bytes 2024-04-06T08:12:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1565877248 bytes 2024-04-06T08:12:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1602790400 bytes 2024-04-06T08:12:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1646295040 bytes 2024-04-06T08:12:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1709801472 bytes 2024-04-06T08:12:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1760640000 bytes 2024-04-06T08:12:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1787678720 bytes 2024-04-06T08:12:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1831554048 bytes 2024-04-06T08:12:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1925823488 bytes 2024-04-06T08:12:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1963682816 bytes 2024-04-06T08:12:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2043470848 bytes 2024-04-06T08:12:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2099316736 bytes 2024-04-06T08:12:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2128185344 bytes 2024-04-06T08:12:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2209110016 bytes 2024-04-06T08:12:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2238690304 bytes 2024-04-06T08:12:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2295269376 bytes 2024-04-06T08:12:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2344856576 bytes 2024-04-06T08:12:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2358097920 bytes 2024-04-06T08:12:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2367858688 bytes 2024-04-06T08:12:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2406985728 bytes 2024-04-06T08:12:11.047 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8718894349584118) > weight (0.25) 2024-04-06T08:12:11.047 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8584325669788067) > weight (0.25) 2024-04-06T08:12:11.047 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T08:12:11.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T08:12:11.048 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-06T08:12:11.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:12:11.046+0000 7f1cc7bd6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 37020) UID: 1000 2024-04-06T08:12:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2515340288 bytes 2024-04-06T08:12:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2538075136 bytes 2024-04-06T08:12:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2556523520 bytes 2024-04-06T08:12:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2589512704 bytes 2024-04-06T08:12:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2590154752 bytes 2024-04-06T08:12:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2660464640 bytes 2024-04-06T08:12:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2703617024 bytes 2024-04-06T08:12:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2784445440 bytes 2024-04-06T08:12:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2838232064 bytes 2024-04-06T08:12:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2855768064 bytes 2024-04-06T08:12:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2932374528 bytes 2024-04-06T08:12:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2970191872 bytes 2024-04-06T08:12:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2975384576 bytes 2024-04-06T08:12:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 3035400192 bytes 2024-04-06T08:12:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 3070899200 bytes 2024-04-06T08:12:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 3080895488 bytes 2024-04-06T08:12:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:12:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:12:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:12:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:12:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_31 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_31 128M 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2024-04-06T08:12:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:12:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-04-06T08:12:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:12:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-04-06T08:12:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:12:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-04-06T08:12:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_31': 2024-04-06T08:12:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:12:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:12:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 139a6dfb9020 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.139a6dfb9020 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:13 2024 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:13 2024 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:13 2024 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: a12859fc-49ad-4646-9f39-4212f7cb141b 2024-04-06T08:12:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:12:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:12:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_31 2024-04-06T08:12:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:12:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:12:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:12:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:12:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:12:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 620544 bytes 2024-04-06T08:12:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 39260160 bytes 2024-04-06T08:12:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 100469760 bytes 2024-04-06T08:12:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 150943744 bytes 2024-04-06T08:12:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 151555072 bytes 2024-04-06T08:12:15.950 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T08:12:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T08:12:15.950 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T08:12:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:12:15.950+0000 7f7c17b32640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39101) UID: 1000 2024-04-06T08:12:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 191772672 bytes 2024-04-06T08:12:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 236095488 bytes 2024-04-06T08:12:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 266802176 bytes 2024-04-06T08:12:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 335258624 bytes 2024-04-06T08:12:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 375283712 bytes 2024-04-06T08:12:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 439946240 bytes 2024-04-06T08:12:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 531765248 bytes 2024-04-06T08:12:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 577574912 bytes 2024-04-06T08:12:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 636900352 bytes 2024-04-06T08:12:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 668611584 bytes 2024-04-06T08:12:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 732662784 bytes 2024-04-06T08:12:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 799097856 bytes 2024-04-06T08:12:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 846929920 bytes 2024-04-06T08:12:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 928439296 bytes 2024-04-06T08:12:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 973794304 bytes 2024-04-06T08:12:19.048 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T08:12:19.048 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:12:19.049 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T08:12:19.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T08:12:19.049 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T08:12:19.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T08:12:19.052 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-04-06T08:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 995611648 bytes 2024-04-06T08:12:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1025748992 bytes 2024-04-06T08:12:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1026359296 bytes 2024-04-06T08:12:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1026948096 bytes 2024-04-06T08:12:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1072919552 bytes 2024-04-06T08:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1101991936 bytes 2024-04-06T08:12:19.951 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T08:12:19.951 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:12:19.951 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T08:12:19.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T08:12:19.952 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T08:12:19.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T08:12:19.954 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-04-06T08:12:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1120142336 bytes 2024-04-06T08:12:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1142614016 bytes 2024-04-06T08:12:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1159715840 bytes 2024-04-06T08:12:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1181315072 bytes 2024-04-06T08:12:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1205344256 bytes 2024-04-06T08:12:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1226766336 bytes 2024-04-06T08:12:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1262282752 bytes 2024-04-06T08:12:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1301438464 bytes 2024-04-06T08:12:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1347902464 bytes 2024-04-06T08:12:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1357600768 bytes 2024-04-06T08:12:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1393925120 bytes 2024-04-06T08:12:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1463895040 bytes 2024-04-06T08:12:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1464528896 bytes 2024-04-06T08:12:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1487405056 bytes 2024-04-06T08:12:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1526457344 bytes 2024-04-06T08:12:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1570702336 bytes 2024-04-06T08:12:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1630632960 bytes 2024-04-06T08:12:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1666681856 bytes 2024-04-06T08:12:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1730838528 bytes 2024-04-06T08:12:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1764809728 bytes 2024-04-06T08:12:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1774566400 bytes 2024-04-06T08:12:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1819754496 bytes 2024-04-06T08:12:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1833016320 bytes 2024-04-06T08:12:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1879881728 bytes 2024-04-06T08:12:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1921281024 bytes 2024-04-06T08:12:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1946301440 bytes 2024-04-06T08:12:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2028384256 bytes 2024-04-06T08:12:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2062286848 bytes 2024-04-06T08:12:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2142337024 bytes 2024-04-06T08:12:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2174527488 bytes 2024-04-06T08:12:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2214332416 bytes 2024-04-06T08:12:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2214945792 bytes 2024-04-06T08:12:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2281153536 bytes 2024-04-06T08:12:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2296116224 bytes 2024-04-06T08:12:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2311468032 bytes 2024-04-06T08:12:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2406760448 bytes 2024-04-06T08:12:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2469637120 bytes 2024-04-06T08:12:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2508795904 bytes 2024-04-06T08:12:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2555831296 bytes 2024-04-06T08:12:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2622716928 bytes 2024-04-06T08:12:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2692508672 bytes 2024-04-06T08:12:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2753337344 bytes 2024-04-06T08:12:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2803535872 bytes 2024-04-06T08:12:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2824296448 bytes 2024-04-06T08:12:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2881410048 bytes 2024-04-06T08:12:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2897128448 bytes 2024-04-06T08:12:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 2975346688 bytes 2024-04-06T08:12:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_32 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_32 128M 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:12:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2024-04-06T08:12:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:12:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-04-06T08:12:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:12:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-04-06T08:12:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_32': 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13b5f6e1927a 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13b5f6e1927a 2024-04-06T08:12:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:28 2024 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:28 2024 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:28 2024 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 079afbf7-3e02-4a0c-b385-2d20983af07c 2024-04-06T08:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:12:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:12:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_32 2024-04-06T08:12:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:12:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:12:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:12:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:12:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:12:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 45952000 bytes 2024-04-06T08:12:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 97804288 bytes 2024-04-06T08:12:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 128124928 bytes 2024-04-06T08:12:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 179249152 bytes 2024-04-06T08:12:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 200570880 bytes 2024-04-06T08:12:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 279532544 bytes 2024-04-06T08:12:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 329582592 bytes 2024-04-06T08:12:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 428430336 bytes 2024-04-06T08:12:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 487705600 bytes 2024-04-06T08:12:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 517340160 bytes 2024-04-06T08:12:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 620082176 bytes 2024-04-06T08:12:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 698321920 bytes 2024-04-06T08:12:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 713135104 bytes 2024-04-06T08:12:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 722522112 bytes 2024-04-06T08:12:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 749175808 bytes 2024-04-06T08:12:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 815400960 bytes 2024-04-06T08:12:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 842368000 bytes 2024-04-06T08:12:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 874300416 bytes 2024-04-06T08:12:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 945241088 bytes 2024-04-06T08:12:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 975540224 bytes 2024-04-06T08:12:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1001208832 bytes 2024-04-06T08:12:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1048309760 bytes 2024-04-06T08:12:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1110293504 bytes 2024-04-06T08:12:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1148716032 bytes 2024-04-06T08:12:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1169733632 bytes 2024-04-06T08:12:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1268277248 bytes 2024-04-06T08:12:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1304384512 bytes 2024-04-06T08:12:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1372985344 bytes 2024-04-06T08:12:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1397111808 bytes 2024-04-06T08:12:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1446578176 bytes 2024-04-06T08:12:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1469808640 bytes 2024-04-06T08:12:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1592075264 bytes 2024-04-06T08:12:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1648026624 bytes 2024-04-06T08:12:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1676316672 bytes 2024-04-06T08:12:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1707843584 bytes 2024-04-06T08:12:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1764615168 bytes 2024-04-06T08:12:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1823121408 bytes 2024-04-06T08:12:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1923960832 bytes 2024-04-06T08:12:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1957688320 bytes 2024-04-06T08:12:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1975133184 bytes 2024-04-06T08:12:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 2007840768 bytes 2024-04-06T08:12:38.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 2069893120 bytes 2024-04-06T08:12:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2130274304 bytes 2024-04-06T08:12:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2203544576 bytes 2024-04-06T08:12:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2258636800 bytes 2024-04-06T08:12:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2317793280 bytes 2024-04-06T08:12:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2370558976 bytes 2024-04-06T08:12:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2406629376 bytes 2024-04-06T08:12:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2459727872 bytes 2024-04-06T08:12:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2529731584 bytes 2024-04-06T08:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2558692352 bytes 2024-04-06T08:12:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2615793664 bytes 2024-04-06T08:12:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2661255168 bytes 2024-04-06T08:12:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2667213824 bytes 2024-04-06T08:12:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2702337024 bytes 2024-04-06T08:12:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2747971584 bytes 2024-04-06T08:12:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2813753344 bytes 2024-04-06T08:12:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2831629312 bytes 2024-04-06T08:12:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2836670464 bytes 2024-04-06T08:12:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2868999168 bytes 2024-04-06T08:12:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2963536896 bytes 2024-04-06T08:12:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 3009370112 bytes 2024-04-06T08:12:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 3040662528 bytes 2024-04-06T08:12:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 3061343232 bytes 2024-04-06T08:12:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_33 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_33 128M 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2024-04-06T08:12:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:12:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-04-06T08:12:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:12:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-04-06T08:12:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:12:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_33': 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13c19b198798 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13c19b198798 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:12:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:42 2024 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:42 2024 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:42 2024 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 43b614ea-0108-4b77-ae7f-5b1e2003fce5 2024-04-06T08:12:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:12:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:12:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_33 2024-04-06T08:12:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:12:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:12:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:12:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:12:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:12:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 24548352 bytes 2024-04-06T08:12:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 51214336 bytes 2024-04-06T08:12:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 73614336 bytes 2024-04-06T08:12:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 139684864 bytes 2024-04-06T08:12:44.627 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:12:44.625+0000 7f5a15134640 -1 reset not still connected to 0x55db8dd9ad00 2024-04-06T08:12:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 238818304 bytes 2024-04-06T08:12:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 261070848 bytes 2024-04-06T08:12:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 306759680 bytes 2024-04-06T08:12:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 362797056 bytes 2024-04-06T08:12:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 373071872 bytes 2024-04-06T08:12:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 374973440 bytes 2024-04-06T08:12:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 454068224 bytes 2024-04-06T08:12:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 534779904 bytes 2024-04-06T08:12:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 535369728 bytes 2024-04-06T08:12:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 568245248 bytes 2024-04-06T08:12:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 597146624 bytes 2024-04-06T08:12:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 614624256 bytes 2024-04-06T08:12:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 669696000 bytes 2024-04-06T08:12:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 706278400 bytes 2024-04-06T08:12:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 752080896 bytes 2024-04-06T08:12:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 809460736 bytes 2024-04-06T08:12:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 876008448 bytes 2024-04-06T08:12:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 897368064 bytes 2024-04-06T08:12:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 929683456 bytes 2024-04-06T08:12:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 974008320 bytes 2024-04-06T08:12:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 979264512 bytes 2024-04-06T08:12:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1060232192 bytes 2024-04-06T08:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1144528896 bytes 2024-04-06T08:12:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1211029504 bytes 2024-04-06T08:12:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1258707968 bytes 2024-04-06T08:12:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1290329088 bytes 2024-04-06T08:12:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1290932224 bytes 2024-04-06T08:12:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1347149824 bytes 2024-04-06T08:12:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1377228800 bytes 2024-04-06T08:12:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1427446784 bytes 2024-04-06T08:12:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1434395648 bytes 2024-04-06T08:12:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1459196928 bytes 2024-04-06T08:12:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1495549952 bytes 2024-04-06T08:12:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1497062400 bytes 2024-04-06T08:12:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1511156736 bytes 2024-04-06T08:12:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1566417920 bytes 2024-04-06T08:12:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1635152896 bytes 2024-04-06T08:12:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1684917248 bytes 2024-04-06T08:12:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1696322560 bytes 2024-04-06T08:12:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1731356672 bytes 2024-04-06T08:12:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1769442304 bytes 2024-04-06T08:12:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1773754368 bytes 2024-04-06T08:12:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1795979264 bytes 2024-04-06T08:12:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1851078656 bytes 2024-04-06T08:12:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1875252224 bytes 2024-04-06T08:12:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1885007872 bytes 2024-04-06T08:12:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1945001984 bytes 2024-04-06T08:12:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1982275584 bytes 2024-04-06T08:12:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2018537472 bytes 2024-04-06T08:12:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2065745920 bytes 2024-04-06T08:12:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2089339904 bytes 2024-04-06T08:12:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2137704448 bytes 2024-04-06T08:12:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2179802112 bytes 2024-04-06T08:12:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2256479232 bytes 2024-04-06T08:12:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2330949632 bytes 2024-04-06T08:12:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2367901696 bytes 2024-04-06T08:12:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2379617280 bytes 2024-04-06T08:12:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2397935616 bytes 2024-04-06T08:12:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2422341632 bytes 2024-04-06T08:12:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2465914880 bytes 2024-04-06T08:12:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2507084800 bytes 2024-04-06T08:12:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2562539520 bytes 2024-04-06T08:12:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2651350016 bytes 2024-04-06T08:12:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2690290688 bytes 2024-04-06T08:12:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2710315008 bytes 2024-04-06T08:12:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2727890944 bytes 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_34 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_34 128M 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:12:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-04-06T08:12:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:12:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_34': 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13cdc50e4b0c 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13cdc50e4b0c 2024-04-06T08:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:56 2024 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:56 2024 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:56 2024 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 6d663376-e7fa-4e9d-b581-641ed5f5b057 2024-04-06T08:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_34 2024-04-06T08:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:12:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 39218176 bytes 2024-04-06T08:12:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 39828480 bytes 2024-04-06T08:12:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 98525184 bytes 2024-04-06T08:12:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 152589312 bytes 2024-04-06T08:12:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 178877440 bytes 2024-04-06T08:12:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 238561280 bytes 2024-04-06T08:12:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 264478720 bytes 2024-04-06T08:12:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 282871808 bytes 2024-04-06T08:12:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 297394176 bytes 2024-04-06T08:12:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 333864960 bytes 2024-04-06T08:13:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 452441088 bytes 2024-04-06T08:13:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 556277760 bytes 2024-04-06T08:13:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 612149248 bytes 2024-04-06T08:13:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 627308544 bytes 2024-04-06T08:13:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 673705984 bytes 2024-04-06T08:13:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 757598208 bytes 2024-04-06T08:13:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 771107840 bytes 2024-04-06T08:13:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 826028032 bytes 2024-04-06T08:13:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 891400192 bytes 2024-04-06T08:13:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 939898880 bytes 2024-04-06T08:13:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 995526656 bytes 2024-04-06T08:13:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1003842560 bytes 2024-04-06T08:13:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1044646912 bytes 2024-04-06T08:13:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1101836288 bytes 2024-04-06T08:13:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1159443456 bytes 2024-04-06T08:13:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1188208640 bytes 2024-04-06T08:13:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1228568576 bytes 2024-04-06T08:13:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1251321856 bytes 2024-04-06T08:13:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1294308352 bytes 2024-04-06T08:13:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1343905792 bytes 2024-04-06T08:13:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1363943424 bytes 2024-04-06T08:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1427017728 bytes 2024-04-06T08:13:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1482165248 bytes 2024-04-06T08:13:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1508433920 bytes 2024-04-06T08:13:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1578671104 bytes 2024-04-06T08:13:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1615849472 bytes 2024-04-06T08:13:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1629873152 bytes 2024-04-06T08:13:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1692323840 bytes 2024-04-06T08:13:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1741108224 bytes 2024-04-06T08:13:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1807931392 bytes 2024-04-06T08:13:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1905137664 bytes 2024-04-06T08:13:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1927191552 bytes 2024-04-06T08:13:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2015929344 bytes 2024-04-06T08:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2083180544 bytes 2024-04-06T08:13:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2101178368 bytes 2024-04-06T08:13:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2122782720 bytes 2024-04-06T08:13:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2202025984 bytes 2024-04-06T08:13:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2224312320 bytes 2024-04-06T08:13:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2227097600 bytes 2024-04-06T08:13:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2269421568 bytes 2024-04-06T08:13:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2351958016 bytes 2024-04-06T08:13:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2378906624 bytes 2024-04-06T08:13:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2379541504 bytes 2024-04-06T08:13:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2406413312 bytes 2024-04-06T08:13:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2429902848 bytes 2024-04-06T08:13:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2477683712 bytes 2024-04-06T08:13:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2494161920 bytes 2024-04-06T08:13:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2499039232 bytes 2024-04-06T08:13:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2561806336 bytes 2024-04-06T08:13:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2573155328 bytes 2024-04-06T08:13:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2654201856 bytes 2024-04-06T08:13:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2733192192 bytes 2024-04-06T08:13:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2757400576 bytes 2024-04-06T08:13:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2812125184 bytes 2024-04-06T08:13:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2875365376 bytes 2024-04-06T08:13:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2945774592 bytes 2024-04-06T08:13:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2996417536 bytes 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_35 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_35 128M 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:13:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-04-06T08:13:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:13:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-04-06T08:13:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_35': 2024-04-06T08:13:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:13:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13d9a176d62f 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13d9a176d62f 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:10 2024 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:10 2024 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:10 2024 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:13:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8219f002-4b2d-40e9-9cd8-ae83bc156dc4 2024-04-06T08:13:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:13:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:13:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_35 2024-04-06T08:13:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:13:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:13:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:13:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:13:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:13:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 30226432 bytes 2024-04-06T08:13:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 46137344 bytes 2024-04-06T08:13:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 53925888 bytes 2024-04-06T08:13:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 95034368 bytes 2024-04-06T08:13:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 116367360 bytes 2024-04-06T08:13:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 197727232 bytes 2024-04-06T08:13:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 248076288 bytes 2024-04-06T08:13:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 331794432 bytes 2024-04-06T08:13:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 361791488 bytes 2024-04-06T08:13:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 409955328 bytes 2024-04-06T08:13:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 430878720 bytes 2024-04-06T08:13:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 431489024 bytes 2024-04-06T08:13:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 488681472 bytes 2024-04-06T08:13:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 525598720 bytes 2024-04-06T08:13:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 651881472 bytes 2024-04-06T08:13:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 719541248 bytes 2024-04-06T08:13:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 792668160 bytes 2024-04-06T08:13:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 801617920 bytes 2024-04-06T08:13:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 851654656 bytes 2024-04-06T08:13:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 887920640 bytes 2024-04-06T08:13:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 914809856 bytes 2024-04-06T08:13:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 939694080 bytes 2024-04-06T08:13:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1008966656 bytes 2024-04-06T08:13:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1088121856 bytes 2024-04-06T08:13:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1130434560 bytes 2024-04-06T08:13:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1206535168 bytes 2024-04-06T08:13:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1244293120 bytes 2024-04-06T08:13:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1262580736 bytes 2024-04-06T08:13:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1291802624 bytes 2024-04-06T08:13:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1313315840 bytes 2024-04-06T08:13:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1343839232 bytes 2024-04-06T08:13:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1384635392 bytes 2024-04-06T08:13:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1385252864 bytes 2024-04-06T08:13:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1401356288 bytes 2024-04-06T08:13:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1453230080 bytes 2024-04-06T08:13:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1502183424 bytes 2024-04-06T08:13:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1539609600 bytes 2024-04-06T08:13:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1601415168 bytes 2024-04-06T08:13:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1636984832 bytes 2024-04-06T08:13:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1667244032 bytes 2024-04-06T08:13:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1691299840 bytes 2024-04-06T08:13:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1733423104 bytes 2024-04-06T08:13:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1782994944 bytes 2024-04-06T08:13:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1854609408 bytes 2024-04-06T08:13:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1865574400 bytes 2024-04-06T08:13:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1928796160 bytes 2024-04-06T08:13:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2013106176 bytes 2024-04-06T08:13:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2078633984 bytes 2024-04-06T08:13:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2091887616 bytes 2024-04-06T08:13:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2162778112 bytes 2024-04-06T08:13:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2187866112 bytes 2024-04-06T08:13:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2202774528 bytes 2024-04-06T08:13:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2230813696 bytes 2024-04-06T08:13:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2285729792 bytes 2024-04-06T08:13:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2374515712 bytes 2024-04-06T08:13:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2388183040 bytes 2024-04-06T08:13:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2419452928 bytes 2024-04-06T08:13:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2497720320 bytes 2024-04-06T08:13:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2562111488 bytes 2024-04-06T08:13:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2562732032 bytes 2024-04-06T08:13:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2612451328 bytes 2024-04-06T08:13:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2649713664 bytes 2024-04-06T08:13:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_36 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_36 128M 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-04-06T08:13:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:13:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-04-06T08:13:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_36': 2024-04-06T08:13:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:13:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:13:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13e5e3e78edf 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13e5e3e78edf 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:24 2024 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:24 2024 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:24 2024 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:13:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:13:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 111e48f6-921e-4a32-9d71-4f87dd955b5e 2024-04-06T08:13:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_36 2024-04-06T08:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:13:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:13:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:13:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:13:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 94057472 bytes 2024-04-06T08:13:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 141411328 bytes 2024-04-06T08:13:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 205437952 bytes 2024-04-06T08:13:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 232489984 bytes 2024-04-06T08:13:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 286754816 bytes 2024-04-06T08:13:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 317897728 bytes 2024-04-06T08:13:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 401410048 bytes 2024-04-06T08:13:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 419332096 bytes 2024-04-06T08:13:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 438823936 bytes 2024-04-06T08:13:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 462034944 bytes 2024-04-06T08:13:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 547066880 bytes 2024-04-06T08:13:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 550795264 bytes 2024-04-06T08:13:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 551415808 bytes 2024-04-06T08:13:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 552994816 bytes 2024-04-06T08:13:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 580000768 bytes 2024-04-06T08:13:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 580602880 bytes 2024-04-06T08:13:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 635715584 bytes 2024-04-06T08:13:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 666894336 bytes 2024-04-06T08:13:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 714306560 bytes 2024-04-06T08:13:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 751899648 bytes 2024-04-06T08:13:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 834478080 bytes 2024-04-06T08:13:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 928424960 bytes 2024-04-06T08:13:29.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1001814016 bytes 2024-04-06T08:13:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1005146112 bytes 2024-04-06T08:13:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1069903872 bytes 2024-04-06T08:13:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1120164864 bytes 2024-04-06T08:13:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1126100992 bytes 2024-04-06T08:13:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1147699200 bytes 2024-04-06T08:13:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1183149056 bytes 2024-04-06T08:13:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1208896512 bytes 2024-04-06T08:13:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1235160064 bytes 2024-04-06T08:13:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1293530112 bytes 2024-04-06T08:13:31.053 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T08:13:31.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T08:13:31.053 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T08:13:31.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:13:31.052+0000 7ff5b2009640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 32482) UID: 1000 2024-04-06T08:13:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1309132800 bytes 2024-04-06T08:13:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1346705408 bytes 2024-04-06T08:13:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1413327872 bytes 2024-04-06T08:13:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1482915840 bytes 2024-04-06T08:13:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1494907904 bytes 2024-04-06T08:13:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1527684096 bytes 2024-04-06T08:13:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1553281024 bytes 2024-04-06T08:13:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1559668736 bytes 2024-04-06T08:13:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1582453760 bytes 2024-04-06T08:13:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1640773632 bytes 2024-04-06T08:13:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1697818624 bytes 2024-04-06T08:13:33.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1738255360 bytes 2024-04-06T08:13:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1776826368 bytes 2024-04-06T08:13:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1822652416 bytes 2024-04-06T08:13:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1830611968 bytes 2024-04-06T08:13:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1876898816 bytes 2024-04-06T08:13:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1901326336 bytes 2024-04-06T08:13:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1991841792 bytes 2024-04-06T08:13:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2021832704 bytes 2024-04-06T08:13:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2040058880 bytes 2024-04-06T08:13:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2082807808 bytes 2024-04-06T08:13:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2169061376 bytes 2024-04-06T08:13:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2187488256 bytes 2024-04-06T08:13:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2201265152 bytes 2024-04-06T08:13:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2244066304 bytes 2024-04-06T08:13:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2257067008 bytes 2024-04-06T08:13:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2355861504 bytes 2024-04-06T08:13:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2431217664 bytes 2024-04-06T08:13:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2463535104 bytes 2024-04-06T08:13:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2510527488 bytes 2024-04-06T08:13:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2522836992 bytes 2024-04-06T08:13:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2594282496 bytes 2024-04-06T08:13:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:13:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:13:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_37 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_37 128M 2024-04-06T08:13:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2024-04-06T08:13:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:13:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-04-06T08:13:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:13:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-04-06T08:13:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:13:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-04-06T08:13:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_37': 2024-04-06T08:13:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:13:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:13:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 13f08db90618 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.13f08db90618 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:36 2024 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:36 2024 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:36 2024 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 9c0cee78-3594-4b38-aea6-9da642e815a7 2024-04-06T08:13:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:13:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:13:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_37 2024-04-06T08:13:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:13:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:13:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=9 2024-04-06T08:13:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:13:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:13:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 57106432 bytes 2024-04-06T08:13:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 83582976 bytes 2024-04-06T08:13:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 143031296 bytes 2024-04-06T08:13:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 186607616 bytes 2024-04-06T08:13:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 203130880 bytes 2024-04-06T08:13:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 252132352 bytes 2024-04-06T08:13:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 315018240 bytes 2024-04-06T08:13:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 337751040 bytes 2024-04-06T08:13:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 418640896 bytes 2024-04-06T08:13:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 433083392 bytes 2024-04-06T08:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 462058496 bytes 2024-04-06T08:13:40.054 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T08:13:40.055 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:13:40.055 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T08:13:40.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T08:13:40.055 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T08:13:40.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T08:13:40.059 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-04-06T08:13:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 523933696 bytes 2024-04-06T08:13:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 589272064 bytes 2024-04-06T08:13:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 655417344 bytes 2024-04-06T08:13:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 676089856 bytes 2024-04-06T08:13:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 715267072 bytes 2024-04-06T08:13:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 755399680 bytes 2024-04-06T08:13:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 782871552 bytes 2024-04-06T08:13:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 814452736 bytes 2024-04-06T08:13:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 865104896 bytes 2024-04-06T08:13:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 875312128 bytes 2024-04-06T08:13:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 904700928 bytes 2024-04-06T08:13:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 905829376 bytes 2024-04-06T08:13:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 932683776 bytes 2024-04-06T08:13:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 972096512 bytes 2024-04-06T08:13:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1013636096 bytes 2024-04-06T08:13:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1088040960 bytes 2024-04-06T08:13:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1190131712 bytes 2024-04-06T08:13:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1206279168 bytes 2024-04-06T08:13:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1272576000 bytes 2024-04-06T08:13:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1310360576 bytes 2024-04-06T08:13:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1310970880 bytes 2024-04-06T08:13:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1319940096 bytes 2024-04-06T08:13:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1327400960 bytes 2024-04-06T08:13:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1384395776 bytes 2024-04-06T08:13:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1430131712 bytes 2024-04-06T08:13:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1501997056 bytes 2024-04-06T08:13:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1548748800 bytes 2024-04-06T08:13:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1575127040 bytes 2024-04-06T08:13:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1600720896 bytes 2024-04-06T08:13:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1634623488 bytes 2024-04-06T08:13:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1664195584 bytes 2024-04-06T08:13:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1714112512 bytes 2024-04-06T08:13:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1718766592 bytes 2024-04-06T08:13:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1791541248 bytes 2024-04-06T08:13:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1852834816 bytes 2024-04-06T08:13:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1957210112 bytes 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_38 2024-04-06T08:13:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_38 128M 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:13:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:13:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:13:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:13:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:13:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2024-04-06T08:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-04-06T08:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:13:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:13:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-04-06T08:13:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:13:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_38': 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 140373ee0923 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.140373ee0923 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:13:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:46 2024 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:46 2024 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:46 2024 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: bee07777-cb98-40aa-a6a6-2b3e5d90cb96 2024-04-06T08:13:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:13:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:13:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_38 2024-04-06T08:13:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:13:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:13:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:13:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:13:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:13:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:13:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:13:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 73756672 bytes 2024-04-06T08:13:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 124831744 bytes 2024-04-06T08:13:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 157957120 bytes 2024-04-06T08:13:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 172808192 bytes 2024-04-06T08:13:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 208461824 bytes 2024-04-06T08:13:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 209087488 bytes 2024-04-06T08:13:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 251191296 bytes 2024-04-06T08:13:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 337065984 bytes 2024-04-06T08:13:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 417589248 bytes 2024-04-06T08:13:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 440478720 bytes 2024-04-06T08:13:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 471380992 bytes 2024-04-06T08:13:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 520453120 bytes 2024-04-06T08:13:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 526371840 bytes 2024-04-06T08:13:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 574713856 bytes 2024-04-06T08:13:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 600435712 bytes 2024-04-06T08:13:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 635433984 bytes 2024-04-06T08:13:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 644967424 bytes 2024-04-06T08:13:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 737661952 bytes 2024-04-06T08:13:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 783776768 bytes 2024-04-06T08:13:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 860588032 bytes 2024-04-06T08:13:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 930946048 bytes 2024-04-06T08:13:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 960797696 bytes 2024-04-06T08:13:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 993434624 bytes 2024-04-06T08:13:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1036473344 bytes 2024-04-06T08:13:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1053715456 bytes 2024-04-06T08:13:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1188097024 bytes 2024-04-06T08:13:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1232036864 bytes 2024-04-06T08:13:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1256898560 bytes 2024-04-06T08:13:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1260019712 bytes 2024-04-06T08:13:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1297998848 bytes 2024-04-06T08:13:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1319651328 bytes 2024-04-06T08:13:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1320249344 bytes 2024-04-06T08:13:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1370194944 bytes 2024-04-06T08:13:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1427766272 bytes 2024-04-06T08:13:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1496717312 bytes 2024-04-06T08:13:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1515347968 bytes 2024-04-06T08:13:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1543502848 bytes 2024-04-06T08:13:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1627823104 bytes 2024-04-06T08:13:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1702455296 bytes 2024-04-06T08:13:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1723715584 bytes 2024-04-06T08:13:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1801743360 bytes 2024-04-06T08:13:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1802340352 bytes 2024-04-06T08:13:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1871124480 bytes 2024-04-06T08:13:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1900359680 bytes 2024-04-06T08:13:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1915847680 bytes 2024-04-06T08:13:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1923825664 bytes 2024-04-06T08:13:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1975162880 bytes 2024-04-06T08:13:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2014480384 bytes 2024-04-06T08:13:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2061348864 bytes 2024-04-06T08:13:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2134795264 bytes 2024-04-06T08:13:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2220093440 bytes 2024-04-06T08:13:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2281530368 bytes 2024-04-06T08:13:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2319242240 bytes 2024-04-06T08:13:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2386692096 bytes 2024-04-06T08:13:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2412129280 bytes 2024-04-06T08:13:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2425669632 bytes 2024-04-06T08:13:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2475738112 bytes 2024-04-06T08:13:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2522268672 bytes 2024-04-06T08:13:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2541203456 bytes 2024-04-06T08:13:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2576987136 bytes 2024-04-06T08:13:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2649393152 bytes 2024-04-06T08:13:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2695640064 bytes 2024-04-06T08:13:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2721120256 bytes 2024-04-06T08:14:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2768901120 bytes 2024-04-06T08:14:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2810049536 bytes 2024-04-06T08:14:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2860850176 bytes 2024-04-06T08:14:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2915819520 bytes 2024-04-06T08:14:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2946638848 bytes 2024-04-06T08:14:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 3009705984 bytes 2024-04-06T08:14:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 3056080896 bytes 2024-04-06T08:14:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 3084122112 bytes 2024-04-06T08:14:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_39 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_39 128M 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2024-04-06T08:14:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-04-06T08:14:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:14:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-04-06T08:14:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-04-06T08:14:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_39': 2024-04-06T08:14:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 140f2b5086c1 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.140f2b5086c1 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:01 2024 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:01 2024 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:01 2024 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 2f88af95-4f57-486d-93b3-f203c018dd1a 2024-04-06T08:14:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_39 2024-04-06T08:14:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:14:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=6 2024-04-06T08:14:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 98076672 bytes 2024-04-06T08:14:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 122260480 bytes 2024-04-06T08:14:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 181704704 bytes 2024-04-06T08:14:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 226287616 bytes 2024-04-06T08:14:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 295948288 bytes 2024-04-06T08:14:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 350512128 bytes 2024-04-06T08:14:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 406116352 bytes 2024-04-06T08:14:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 452877312 bytes 2024-04-06T08:14:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 485196800 bytes 2024-04-06T08:14:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 487544832 bytes 2024-04-06T08:14:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 588615680 bytes 2024-04-06T08:14:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 622439424 bytes 2024-04-06T08:14:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 667184128 bytes 2024-04-06T08:14:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 710725632 bytes 2024-04-06T08:14:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 746428416 bytes 2024-04-06T08:14:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 789677056 bytes 2024-04-06T08:14:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 834666496 bytes 2024-04-06T08:14:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 847804416 bytes 2024-04-06T08:14:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 869328896 bytes 2024-04-06T08:14:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 890398720 bytes 2024-04-06T08:14:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 961177600 bytes 2024-04-06T08:14:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1056445440 bytes 2024-04-06T08:14:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1075092480 bytes 2024-04-06T08:14:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1130395648 bytes 2024-04-06T08:14:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1229111296 bytes 2024-04-06T08:14:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1291121664 bytes 2024-04-06T08:14:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1317185536 bytes 2024-04-06T08:14:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1355020288 bytes 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_40 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_40 128M 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:14:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2024-04-06T08:14:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-04-06T08:14:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:14:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-04-06T08:14:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_40': 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 141b29e2bcfb 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.141b29e2bcfb 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:07 2024 2024-04-06T08:14:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:07 2024 2024-04-06T08:14:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:07 2024 2024-04-06T08:14:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 3436f7ad-064f-4e05-bbf9-791ed79dfc9d 2024-04-06T08:14:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_40 2024-04-06T08:14:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:14:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=6 2024-04-06T08:14:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 73706496 bytes 2024-04-06T08:14:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 74892288 bytes 2024-04-06T08:14:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 93284352 bytes 2024-04-06T08:14:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 151645184 bytes 2024-04-06T08:14:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 209581056 bytes 2024-04-06T08:14:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 218177536 bytes 2024-04-06T08:14:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 275203072 bytes 2024-04-06T08:14:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 293031936 bytes 2024-04-06T08:14:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 334867456 bytes 2024-04-06T08:14:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 353060864 bytes 2024-04-06T08:14:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 402531328 bytes 2024-04-06T08:14:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 445648896 bytes 2024-04-06T08:14:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 487720960 bytes 2024-04-06T08:14:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 515061760 bytes 2024-04-06T08:14:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 538920960 bytes 2024-04-06T08:14:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 587329536 bytes 2024-04-06T08:14:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 610160640 bytes 2024-04-06T08:14:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 643559424 bytes 2024-04-06T08:14:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 683441152 bytes 2024-04-06T08:14:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 697947136 bytes 2024-04-06T08:14:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 729240576 bytes 2024-04-06T08:14:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 783388672 bytes 2024-04-06T08:14:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 824070144 bytes 2024-04-06T08:14:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 835830784 bytes 2024-04-06T08:14:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 898952192 bytes 2024-04-06T08:14:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 923703296 bytes 2024-04-06T08:14:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 985168896 bytes 2024-04-06T08:14:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 989368320 bytes 2024-04-06T08:14:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1056381952 bytes 2024-04-06T08:14:14.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1058841600 bytes 2024-04-06T08:14:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1151546368 bytes 2024-04-06T08:14:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1160438784 bytes 2024-04-06T08:14:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1193537536 bytes 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_41 2024-04-06T08:14:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_41 128M 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:14:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2024-04-06T08:14:14.954 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-06T08:14:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-06T08:14:14.955 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T08:14:14.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:14:14.951+0000 7fbc88457640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40103) UID: 1000 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-04-06T08:14:15.002 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:14:14.999+0000 7f5a15134640 -1 reset not still connected to 0x55db89d0edd0 2024-04-06T08:14:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-04-06T08:14:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_41': 2024-04-06T08:14:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1427f60698b4 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1427f60698b4 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:14 2024 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:14 2024 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:14 2024 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 175e0311-52c5-4f3f-9e73-a36267e32a86 2024-04-06T08:14:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_41 2024-04-06T08:14:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:14:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T08:14:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 37336064 bytes 2024-04-06T08:14:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 66577408 bytes 2024-04-06T08:14:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 122813440 bytes 2024-04-06T08:14:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 191700992 bytes 2024-04-06T08:14:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 244700160 bytes 2024-04-06T08:14:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 273807360 bytes 2024-04-06T08:14:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 342708224 bytes 2024-04-06T08:14:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 346617856 bytes 2024-04-06T08:14:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 383963136 bytes 2024-04-06T08:14:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 384546816 bytes 2024-04-06T08:14:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 416335872 bytes 2024-04-06T08:14:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 453776384 bytes 2024-04-06T08:14:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 476059648 bytes 2024-04-06T08:14:18.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 539612160 bytes 2024-04-06T08:14:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 555447296 bytes 2024-04-06T08:14:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 574775296 bytes 2024-04-06T08:14:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 644044800 bytes 2024-04-06T08:14:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 675216384 bytes 2024-04-06T08:14:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 702190592 bytes 2024-04-06T08:14:18.955 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-06T08:14:18.955 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:14:18.956 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-06T08:14:18.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-06T08:14:18.956 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-06T08:14:18.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-06T08:14:18.959 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-04-06T08:14:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 778582016 bytes 2024-04-06T08:14:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 831891456 bytes 2024-04-06T08:14:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 908867584 bytes 2024-04-06T08:14:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 931723264 bytes 2024-04-06T08:14:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 985361408 bytes 2024-04-06T08:14:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1043072000 bytes 2024-04-06T08:14:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1066147840 bytes 2024-04-06T08:14:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1109989376 bytes 2024-04-06T08:14:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1139610624 bytes 2024-04-06T08:14:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1183631360 bytes 2024-04-06T08:14:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1233721344 bytes 2024-04-06T08:14:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1292790784 bytes 2024-04-06T08:14:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1303897088 bytes 2024-04-06T08:14:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1326491648 bytes 2024-04-06T08:14:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1378105344 bytes 2024-04-06T08:14:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1391069184 bytes 2024-04-06T08:14:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1442435072 bytes 2024-04-06T08:14:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1530142720 bytes 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_42 2024-04-06T08:14:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_42 128M 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:14:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2024-04-06T08:14:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-04-06T08:14:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:14:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:22.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-04-06T08:14:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-04-06T08:14:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_42': 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 143961aaf25f 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.143961aaf25f 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:22 2024 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:22 2024 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:22 2024 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f200c959-0115-400f-90ae-1785585fdbef 2024-04-06T08:14:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_42 2024-04-06T08:14:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:14:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=10 2024-04-06T08:14:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 25260032 bytes 2024-04-06T08:14:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 46663680 bytes 2024-04-06T08:14:23.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 78190592 bytes 2024-04-06T08:14:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 111339520 bytes 2024-04-06T08:14:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 144415744 bytes 2024-04-06T08:14:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 226376704 bytes 2024-04-06T08:14:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 256776192 bytes 2024-04-06T08:14:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 274479104 bytes 2024-04-06T08:14:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 356997120 bytes 2024-04-06T08:14:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 389911552 bytes 2024-04-06T08:14:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 455284736 bytes 2024-04-06T08:14:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 479309824 bytes 2024-04-06T08:14:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 512940032 bytes 2024-04-06T08:14:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 563969024 bytes 2024-04-06T08:14:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 581268480 bytes 2024-04-06T08:14:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 643997696 bytes 2024-04-06T08:14:25.938 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:14:25.939+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d384f110 2024-04-06T08:14:25.939 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:14:25.939+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d3ee5790 2024-04-06T08:14:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 719109120 bytes 2024-04-06T08:14:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 740552704 bytes 2024-04-06T08:14:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 794255360 bytes 2024-04-06T08:14:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 863942656 bytes 2024-04-06T08:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 910602240 bytes 2024-04-06T08:14:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 941420544 bytes 2024-04-06T08:14:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1025157120 bytes 2024-04-06T08:14:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1074917376 bytes 2024-04-06T08:14:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1120859136 bytes 2024-04-06T08:14:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1140468736 bytes 2024-04-06T08:14:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1214661632 bytes 2024-04-06T08:14:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1250727936 bytes 2024-04-06T08:14:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1260894208 bytes 2024-04-06T08:14:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1312237568 bytes 2024-04-06T08:14:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1372450816 bytes 2024-04-06T08:14:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1387439104 bytes 2024-04-06T08:14:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1446476800 bytes 2024-04-06T08:14:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1513437184 bytes 2024-04-06T08:14:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1541345280 bytes 2024-04-06T08:14:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1613069312 bytes 2024-04-06T08:14:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1645921280 bytes 2024-04-06T08:14:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1706997760 bytes 2024-04-06T08:14:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1764883456 bytes 2024-04-06T08:14:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1838131200 bytes 2024-04-06T08:14:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1895243776 bytes 2024-04-06T08:14:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1904377856 bytes 2024-04-06T08:14:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1926239232 bytes 2024-04-06T08:14:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1949116416 bytes 2024-04-06T08:14:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1964279808 bytes 2024-04-06T08:14:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2025771008 bytes 2024-04-06T08:14:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2051785728 bytes 2024-04-06T08:14:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2107561984 bytes 2024-04-06T08:14:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2133547008 bytes 2024-04-06T08:14:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2150902784 bytes 2024-04-06T08:14:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2204593152 bytes 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_43 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-04-06T08:14:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_43 128M 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2024-04-06T08:14:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-04-06T08:14:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:14:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-04-06T08:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_43': 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1445902cd1f7 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1445902cd1f7 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:32 2024 2024-04-06T08:14:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:32 2024 2024-04-06T08:14:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:32 2024 2024-04-06T08:14:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 1a6528d8-9d7a-4762-8686-78b7552a3f94 2024-04-06T08:14:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_43 2024-04-06T08:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:14:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:14:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 24823808 bytes 2024-04-06T08:14:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 61937664 bytes 2024-04-06T08:14:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 114758656 bytes 2024-04-06T08:14:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 200432640 bytes 2024-04-06T08:14:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 242898944 bytes 2024-04-06T08:14:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 285659136 bytes 2024-04-06T08:14:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 373273600 bytes 2024-04-06T08:14:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 413345792 bytes 2024-04-06T08:14:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 463864832 bytes 2024-04-06T08:14:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 502370304 bytes 2024-04-06T08:14:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 561466368 bytes 2024-04-06T08:14:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 640543744 bytes 2024-04-06T08:14:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 651857920 bytes 2024-04-06T08:14:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 722294784 bytes 2024-04-06T08:14:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 739488768 bytes 2024-04-06T08:14:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 788829184 bytes 2024-04-06T08:14:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 846977024 bytes 2024-04-06T08:14:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 899907584 bytes 2024-04-06T08:14:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 900503552 bytes 2024-04-06T08:14:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 915523584 bytes 2024-04-06T08:14:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 929620992 bytes 2024-04-06T08:14:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1046678528 bytes 2024-04-06T08:14:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1091314688 bytes 2024-04-06T08:14:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1169005568 bytes 2024-04-06T08:14:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1192677376 bytes 2024-04-06T08:14:39.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1226494976 bytes 2024-04-06T08:14:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1244955648 bytes 2024-04-06T08:14:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1326325760 bytes 2024-04-06T08:14:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1376635904 bytes 2024-04-06T08:14:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1400155136 bytes 2024-04-06T08:14:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1427421184 bytes 2024-04-06T08:14:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1489346560 bytes 2024-04-06T08:14:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1559863296 bytes 2024-04-06T08:14:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1580953600 bytes 2024-04-06T08:14:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1713034240 bytes 2024-04-06T08:14:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1733916672 bytes 2024-04-06T08:14:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1821360128 bytes 2024-04-06T08:14:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1909566464 bytes 2024-04-06T08:14:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 2004675584 bytes 2024-04-06T08:14:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 2022192128 bytes 2024-04-06T08:14:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 2067076096 bytes 2024-04-06T08:14:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 2089188352 bytes 2024-04-06T08:14:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2111640576 bytes 2024-04-06T08:14:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2112266240 bytes 2024-04-06T08:14:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2145861632 bytes 2024-04-06T08:14:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2195908608 bytes 2024-04-06T08:14:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2229184512 bytes 2024-04-06T08:14:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2278885376 bytes 2024-04-06T08:14:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2405692416 bytes 2024-04-06T08:14:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2478531584 bytes 2024-04-06T08:14:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2510610432 bytes 2024-04-06T08:14:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2522851328 bytes 2024-04-06T08:14:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2537192448 bytes 2024-04-06T08:14:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2539080704 bytes 2024-04-06T08:14:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2547708928 bytes 2024-04-06T08:14:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2588208128 bytes 2024-04-06T08:14:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2606389248 bytes 2024-04-06T08:14:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2665398272 bytes 2024-04-06T08:14:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2725781504 bytes 2024-04-06T08:14:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2789867520 bytes 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_44 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_44 128M 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2024-04-06T08:14:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-04-06T08:14:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:14:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-04-06T08:14:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-04-06T08:14:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_44': 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 145140f062ab 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.145140f062ab 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:45 2024 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:45 2024 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:45 2024 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 79a01d9a-fb10-4dfa-97e4-0874260e4dde 2024-04-06T08:14:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_44 2024-04-06T08:14:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:14:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:14:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 26465280 bytes 2024-04-06T08:14:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 107543552 bytes 2024-04-06T08:14:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 111695872 bytes 2024-04-06T08:14:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 132787200 bytes 2024-04-06T08:14:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 144049152 bytes 2024-04-06T08:14:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 179269632 bytes 2024-04-06T08:14:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 201959424 bytes 2024-04-06T08:14:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 253040640 bytes 2024-04-06T08:14:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 281673728 bytes 2024-04-06T08:14:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 317944832 bytes 2024-04-06T08:14:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 356836352 bytes 2024-04-06T08:14:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 391632896 bytes 2024-04-06T08:14:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 406309888 bytes 2024-04-06T08:14:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 457506816 bytes 2024-04-06T08:14:49.060 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7414846876521425) > weight (0.25) 2024-04-06T08:14:49.060 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-06T08:14:49.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-06T08:14:49.061 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T08:14:49.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:14:49.058+0000 7f2f8172e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 28947) UID: 1000 2024-04-06T08:14:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 495627264 bytes 2024-04-06T08:14:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 546962432 bytes 2024-04-06T08:14:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 637077504 bytes 2024-04-06T08:14:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 665077760 bytes 2024-04-06T08:14:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 726968320 bytes 2024-04-06T08:14:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 757591040 bytes 2024-04-06T08:14:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 758216704 bytes 2024-04-06T08:14:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 801849344 bytes 2024-04-06T08:14:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 820256768 bytes 2024-04-06T08:14:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 863798272 bytes 2024-04-06T08:14:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 930701312 bytes 2024-04-06T08:14:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 962296832 bytes 2024-04-06T08:14:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 974738432 bytes 2024-04-06T08:14:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1008956416 bytes 2024-04-06T08:14:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1064276992 bytes 2024-04-06T08:14:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1064907776 bytes 2024-04-06T08:14:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1099995136 bytes 2024-04-06T08:14:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1208174592 bytes 2024-04-06T08:14:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1237827584 bytes 2024-04-06T08:14:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1244274688 bytes 2024-04-06T08:14:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1263272960 bytes 2024-04-06T08:14:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1285671936 bytes 2024-04-06T08:14:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1332484096 bytes 2024-04-06T08:14:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1350290432 bytes 2024-04-06T08:14:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1433062400 bytes 2024-04-06T08:14:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1491241984 bytes 2024-04-06T08:14:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1561541632 bytes 2024-04-06T08:14:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1582987264 bytes 2024-04-06T08:14:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1639693312 bytes 2024-04-06T08:14:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1753475072 bytes 2024-04-06T08:14:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1761959936 bytes 2024-04-06T08:14:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1803900928 bytes 2024-04-06T08:14:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1828261888 bytes 2024-04-06T08:14:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1897604096 bytes 2024-04-06T08:14:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1943676928 bytes 2024-04-06T08:14:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1968953344 bytes 2024-04-06T08:14:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1978836992 bytes 2024-04-06T08:14:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2026952704 bytes 2024-04-06T08:14:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2085635072 bytes 2024-04-06T08:14:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2152735744 bytes 2024-04-06T08:14:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2218549248 bytes 2024-04-06T08:14:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2291201024 bytes 2024-04-06T08:14:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2394177536 bytes 2024-04-06T08:14:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2465423360 bytes 2024-04-06T08:14:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2479383552 bytes 2024-04-06T08:14:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2480018432 bytes 2024-04-06T08:14:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2489160704 bytes 2024-04-06T08:14:58.062 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-06T08:14:58.062 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:14:58.062 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-06T08:14:58.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T08:14:58.063 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T08:14:58.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T08:14:58.065 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-06T08:14:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2524765184 bytes 2024-04-06T08:14:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2543852544 bytes 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_45 2024-04-06T08:14:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_45 128M 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:14:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2024-04-06T08:14:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:14:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-04-06T08:14:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:14:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:14:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-04-06T08:14:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:14:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-04-06T08:14:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_45': 2024-04-06T08:14:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:14:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:14:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:14:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 14654142e33a 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.14654142e33a 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:58 2024 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:58 2024 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:58 2024 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 5ba3668d-93d2-4d22-87c3-f63064c87646 2024-04-06T08:14:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:14:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:14:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_45 2024-04-06T08:14:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:14:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:14:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:14:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:14:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=6 2024-04-06T08:14:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:14:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:14:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 32079872 bytes 2024-04-06T08:14:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 63219712 bytes 2024-04-06T08:14:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 84714496 bytes 2024-04-06T08:14:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 114188288 bytes 2024-04-06T08:14:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 127628288 bytes 2024-04-06T08:15:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 197177344 bytes 2024-04-06T08:15:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 197787648 bytes 2024-04-06T08:15:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 269719552 bytes 2024-04-06T08:15:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 299556864 bytes 2024-04-06T08:15:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 381360128 bytes 2024-04-06T08:15:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 420378624 bytes 2024-04-06T08:15:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 457796608 bytes 2024-04-06T08:15:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 496230400 bytes 2024-04-06T08:15:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 612086784 bytes 2024-04-06T08:15:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 641618944 bytes 2024-04-06T08:15:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 692366336 bytes 2024-04-06T08:15:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 758907904 bytes 2024-04-06T08:15:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 784689152 bytes 2024-04-06T08:15:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 861054976 bytes 2024-04-06T08:15:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 871273472 bytes 2024-04-06T08:15:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 875082752 bytes 2024-04-06T08:15:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 906059776 bytes 2024-04-06T08:15:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 957113344 bytes 2024-04-06T08:15:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 977318912 bytes 2024-04-06T08:15:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 989283328 bytes 2024-04-06T08:15:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 994804736 bytes 2024-04-06T08:15:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1017286656 bytes 2024-04-06T08:15:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1034374144 bytes 2024-04-06T08:15:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1044482048 bytes 2024-04-06T08:15:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1056259072 bytes 2024-04-06T08:15:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1095690240 bytes 2024-04-06T08:15:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1128879104 bytes 2024-04-06T08:15:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1196804096 bytes 2024-04-06T08:15:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:15:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:15:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:15:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:15:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_46 2024-04-06T08:15:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-04-06T08:15:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_46 128M 2024-04-06T08:15:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:15:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:15:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:15:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:15:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2024-04-06T08:15:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:15:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-04-06T08:15:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:15:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-04-06T08:15:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:15:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_46': 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1471ef4afbdf 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1471ef4afbdf 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:05 2024 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:05 2024 2024-04-06T08:15:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:05 2024 2024-04-06T08:15:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:15:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:15:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 5084cde1-4c2c-469a-b871-05d78b2a8610 2024-04-06T08:15:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:15:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:15:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_46 2024-04-06T08:15:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:15:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:15:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=8 2024-04-06T08:15:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:15:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:15:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 36019200 bytes 2024-04-06T08:15:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 71506944 bytes 2024-04-06T08:15:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 136855552 bytes 2024-04-06T08:15:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 201696256 bytes 2024-04-06T08:15:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 229925888 bytes 2024-04-06T08:15:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 230523904 bytes 2024-04-06T08:15:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 311851008 bytes 2024-04-06T08:15:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 368013312 bytes 2024-04-06T08:15:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 406981632 bytes 2024-04-06T08:15:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 464624640 bytes 2024-04-06T08:15:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 507312128 bytes 2024-04-06T08:15:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 507922432 bytes 2024-04-06T08:15:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 549553152 bytes 2024-04-06T08:15:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 593363968 bytes 2024-04-06T08:15:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 664286208 bytes 2024-04-06T08:15:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 708614144 bytes 2024-04-06T08:15:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 722308096 bytes 2024-04-06T08:15:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 752850944 bytes 2024-04-06T08:15:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 777410560 bytes 2024-04-06T08:15:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 811119616 bytes 2024-04-06T08:15:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 871854080 bytes 2024-04-06T08:15:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 964384768 bytes 2024-04-06T08:15:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 997634048 bytes 2024-04-06T08:15:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1062323200 bytes 2024-04-06T08:15:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1117702144 bytes 2024-04-06T08:15:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1128381440 bytes 2024-04-06T08:15:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1157656576 bytes 2024-04-06T08:15:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1208948736 bytes 2024-04-06T08:15:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1239018496 bytes 2024-04-06T08:15:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1304401920 bytes 2024-04-06T08:15:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1327288320 bytes 2024-04-06T08:15:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1372571648 bytes 2024-04-06T08:15:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1447416832 bytes 2024-04-06T08:15:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1471719424 bytes 2024-04-06T08:15:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1485092864 bytes 2024-04-06T08:15:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1497279488 bytes 2024-04-06T08:15:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1497873408 bytes 2024-04-06T08:15:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1498512384 bytes 2024-04-06T08:15:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1533669376 bytes 2024-04-06T08:15:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1583508480 bytes 2024-04-06T08:15:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1616285696 bytes 2024-04-06T08:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1637359616 bytes 2024-04-06T08:15:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1651325952 bytes 2024-04-06T08:15:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_47 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_47 128M 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-04-06T08:15:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:15:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-04-06T08:15:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_47': 2024-04-06T08:15:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:15:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:15:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:15:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 147d1881548 2024-04-06T08:15:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.147d1881548 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:13 2024 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:13 2024 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:13 2024 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 339c1583-8140-4531-ae3a-4bb2efc43b9e 2024-04-06T08:15:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:15:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:15:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_47 2024-04-06T08:15:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:15:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:15:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=6 2024-04-06T08:15:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:15:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:15:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 57163776 bytes 2024-04-06T08:15:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 103493632 bytes 2024-04-06T08:15:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 157898752 bytes 2024-04-06T08:15:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 172268544 bytes 2024-04-06T08:15:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 210564096 bytes 2024-04-06T08:15:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 226479104 bytes 2024-04-06T08:15:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 295810048 bytes 2024-04-06T08:15:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 351802368 bytes 2024-04-06T08:15:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 406545408 bytes 2024-04-06T08:15:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 520735744 bytes 2024-04-06T08:15:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 535023616 bytes 2024-04-06T08:15:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 620733440 bytes 2024-04-06T08:15:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 667395072 bytes 2024-04-06T08:15:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 681914368 bytes 2024-04-06T08:15:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 682543104 bytes 2024-04-06T08:15:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 706295808 bytes 2024-04-06T08:15:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 707124224 bytes 2024-04-06T08:15:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 793257984 bytes 2024-04-06T08:15:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 810449920 bytes 2024-04-06T08:15:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 844078080 bytes 2024-04-06T08:15:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 878952448 bytes 2024-04-06T08:15:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 980030464 bytes 2024-04-06T08:15:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1035786240 bytes 2024-04-06T08:15:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1082884096 bytes 2024-04-06T08:15:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1159986176 bytes 2024-04-06T08:15:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1181841408 bytes 2024-04-06T08:15:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1194849280 bytes 2024-04-06T08:15:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1222140928 bytes 2024-04-06T08:15:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1264719872 bytes 2024-04-06T08:15:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1296951296 bytes 2024-04-06T08:15:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1337947136 bytes 2024-04-06T08:15:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_48 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:15:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_48 128M 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-04-06T08:15:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:15:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-04-06T08:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_48': 2024-04-06T08:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:15:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 14895ee1ec16 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.14895ee1ec16 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:19 2024 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:19 2024 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:19 2024 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ed498893-396c-4438-80fa-19e53e8f4471 2024-04-06T08:15:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:15:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:15:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_48 2024-04-06T08:15:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:15:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:15:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T08:15:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:15:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:15:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 599040 bytes 2024-04-06T08:15:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 47237120 bytes 2024-04-06T08:15:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 71241728 bytes 2024-04-06T08:15:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 95440896 bytes 2024-04-06T08:15:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 160712704 bytes 2024-04-06T08:15:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 179140608 bytes 2024-04-06T08:15:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 179761152 bytes 2024-04-06T08:15:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 188091392 bytes 2024-04-06T08:15:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 247400448 bytes 2024-04-06T08:15:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 319074304 bytes 2024-04-06T08:15:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 346333184 bytes 2024-04-06T08:15:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 361579520 bytes 2024-04-06T08:15:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 373092352 bytes 2024-04-06T08:15:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 399490048 bytes 2024-04-06T08:15:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 421449728 bytes 2024-04-06T08:15:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 461106176 bytes 2024-04-06T08:15:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 464512000 bytes 2024-04-06T08:15:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 477555712 bytes 2024-04-06T08:15:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 493630464 bytes 2024-04-06T08:15:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 528744448 bytes 2024-04-06T08:15:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 556434432 bytes 2024-04-06T08:15:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 601420800 bytes 2024-04-06T08:15:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 602033152 bytes 2024-04-06T08:15:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 619963392 bytes 2024-04-06T08:15:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 653708288 bytes 2024-04-06T08:15:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 701786112 bytes 2024-04-06T08:15:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 747250688 bytes 2024-04-06T08:15:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 777663488 bytes 2024-04-06T08:15:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 812975104 bytes 2024-04-06T08:15:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 888862720 bytes 2024-04-06T08:15:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1009503232 bytes 2024-04-06T08:15:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1037966336 bytes 2024-04-06T08:15:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1050605568 bytes 2024-04-06T08:15:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1085078528 bytes 2024-04-06T08:15:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1150059520 bytes 2024-04-06T08:15:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1179667456 bytes 2024-04-06T08:15:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1220932608 bytes 2024-04-06T08:15:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1259452416 bytes 2024-04-06T08:15:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1277293568 bytes 2024-04-06T08:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:15:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_49 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_49 128M 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2024-04-06T08:15:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:15:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-04-06T08:15:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:15:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-04-06T08:15:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:15:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-04-06T08:15:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_49': 2024-04-06T08:15:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1495e4fa8f0d 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1495e4fa8f0d 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:27 2024 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:27 2024 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:27 2024 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:15:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f62960e8-a1e8-461c-bf37-b3f594ac8f8b 2024-04-06T08:15:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:15:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:15:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_49 2024-04-06T08:15:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:15:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:15:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=11 2024-04-06T08:15:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:15:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:15:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 35482624 bytes 2024-04-06T08:15:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 50840576 bytes 2024-04-06T08:15:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 69984256 bytes 2024-04-06T08:15:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 80833536 bytes 2024-04-06T08:15:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 110652416 bytes 2024-04-06T08:15:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 153079808 bytes 2024-04-06T08:15:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 166902784 bytes 2024-04-06T08:15:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 222747648 bytes 2024-04-06T08:15:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 250919936 bytes 2024-04-06T08:15:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 277183488 bytes 2024-04-06T08:15:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 292015104 bytes 2024-04-06T08:15:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 333597696 bytes 2024-04-06T08:15:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 361846784 bytes 2024-04-06T08:15:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 406594560 bytes 2024-04-06T08:15:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 427914240 bytes 2024-04-06T08:15:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 466391040 bytes 2024-04-06T08:15:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 492324864 bytes 2024-04-06T08:15:31.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 507980800 bytes 2024-04-06T08:15:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 561310720 bytes 2024-04-06T08:15:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 682765312 bytes 2024-04-06T08:15:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 740831232 bytes 2024-04-06T08:15:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 790676480 bytes 2024-04-06T08:15:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 831971328 bytes 2024-04-06T08:15:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 851264512 bytes 2024-04-06T08:15:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 879893504 bytes 2024-04-06T08:15:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 916722688 bytes 2024-04-06T08:15:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 936187904 bytes 2024-04-06T08:15:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 959691776 bytes 2024-04-06T08:15:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1007100928 bytes 2024-04-06T08:15:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1065487360 bytes 2024-04-06T08:15:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1149350912 bytes 2024-04-06T08:15:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1174274048 bytes 2024-04-06T08:15:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1267554304 bytes 2024-04-06T08:15:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1287005184 bytes 2024-04-06T08:15:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1321721856 bytes 2024-04-06T08:15:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1343760384 bytes 2024-04-06T08:15:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1376241664 bytes 2024-04-06T08:15:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1424049152 bytes 2024-04-06T08:15:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1424646144 bytes 2024-04-06T08:15:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1448253440 bytes 2024-04-06T08:15:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1448847360 bytes 2024-04-06T08:15:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1460148224 bytes 2024-04-06T08:15:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1465633792 bytes 2024-04-06T08:15:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1478714368 bytes 2024-04-06T08:15:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1553112064 bytes 2024-04-06T08:15:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1570314240 bytes 2024-04-06T08:15:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1598636032 bytes 2024-04-06T08:15:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1641062400 bytes 2024-04-06T08:15:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1685688320 bytes 2024-04-06T08:15:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1722032128 bytes 2024-04-06T08:15:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1747163136 bytes 2024-04-06T08:15:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1784027136 bytes 2024-04-06T08:15:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 1811314688 bytes 2024-04-06T08:15:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 1835242496 bytes 2024-04-06T08:15:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 1881850880 bytes 2024-04-06T08:15:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 1905090560 bytes 2024-04-06T08:15:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 1925873664 bytes 2024-04-06T08:15:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 1983473664 bytes 2024-04-06T08:15:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 1991564288 bytes 2024-04-06T08:15:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2047357952 bytes 2024-04-06T08:15:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2047972352 bytes 2024-04-06T08:15:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2068294656 bytes 2024-04-06T08:15:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2085114880 bytes 2024-04-06T08:15:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:15:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:15:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:15:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_50 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n snapshot ']' 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_50 128M 2024-04-06T08:15:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 128M ']' 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=128M 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:15:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2024-04-06T08:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' image = image ']' 2024-04-06T08:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-04-06T08:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-04-06T08:15:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:15:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-04-06T08:15:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_50': 2024-04-06T08:15:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:15:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:15:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:15:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 14a1c09081b9 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.14a1c09081b9 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:40 2024 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:40 2024 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:40 2024 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: b626fa42-e982-4b5f-9370-6b8c700fab59 2024-04-06T08:15:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:15:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n '' ']' 2024-04-06T08:15:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_50 2024-04-06T08:15:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:15:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:15:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=6 2024-04-06T08:15:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:15:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:15:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 86447104 bytes 2024-04-06T08:15:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 124085248 bytes 2024-04-06T08:15:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 136160256 bytes 2024-04-06T08:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 155585536 bytes 2024-04-06T08:15:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 181678080 bytes 2024-04-06T08:15:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 188253184 bytes 2024-04-06T08:15:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 282646528 bytes 2024-04-06T08:15:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 310161408 bytes 2024-04-06T08:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 319836160 bytes 2024-04-06T08:15:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 357528576 bytes 2024-04-06T08:15:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 373265408 bytes 2024-04-06T08:15:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 398618624 bytes 2024-04-06T08:15:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 399215616 bytes 2024-04-06T08:15:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 471906304 bytes 2024-04-06T08:15:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 528733184 bytes 2024-04-06T08:15:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 536854528 bytes 2024-04-06T08:15:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 613571584 bytes 2024-04-06T08:15:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 640842752 bytes 2024-04-06T08:15:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 662758400 bytes 2024-04-06T08:15:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 694355968 bytes 2024-04-06T08:15:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 858544128 bytes 2024-04-06T08:15:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 935980032 bytes 2024-04-06T08:15:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 987239424 bytes 2024-04-06T08:15:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 990373888 bytes 2024-04-06T08:15:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1045960704 bytes 2024-04-06T08:15:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1126232064 bytes 2024-04-06T08:15:45.959 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7141378338310753) > weight (0.25) 2024-04-06T08:15:45.959 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9948064726894819) > weight (0.25) 2024-04-06T08:15:45.959 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4387000397230052) > weight (0.25) 2024-04-06T08:15:45.959 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T08:15:45.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T08:15:45.960 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T08:15:45.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:15:45.957+0000 7f6a881f8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 37977) UID: 1000 2024-04-06T08:15:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1195686912 bytes 2024-04-06T08:15:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1221709824 bytes 2024-04-06T08:15:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1258685440 bytes 2024-04-06T08:15:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1295284224 bytes 2024-04-06T08:15:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1311981568 bytes 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_count=50 2024-04-06T08:15:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:15:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local count 2024-04-06T08:15:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_count=0 2024-04-06T08:15:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:15:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 40 2024-04-06T08:15:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:15:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 1 -ne 1 2024-04-06T08:15:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T08:15:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'images: ' 2024-04-06T08:15:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 2 2024-04-06T08:15:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ count=50 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 50 = 50 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state 2024-04-06T08:15:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 40 2024-04-06T08:15:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:15:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 1 -ne 1 2024-04-06T08:15:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T08:15:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'image health:' 2024-04-06T08:15:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 3 2024-04-06T08:15:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:15:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=WARNING 2024-04-06T08:15:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test WARNING = ERROR 2024-04-06T08:15:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test WARNING = OK 2024-04-06T08:15:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:15:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 2 -ne 1 2024-04-06T08:15:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 30 2024-04-06T08:15:48.961 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T08:15:48.961 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:15:48.962 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T08:15:48.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T08:15:48.962 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T08:15:48.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T08:15:48.966 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-04-06T08:16:00.066 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.31553560329575725) > weight (0.25) 2024-04-06T08:16:00.066 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7795795294307627) > weight (0.25) 2024-04-06T08:16:00.067 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T08:16:00.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T08:16:00.067 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T08:16:00.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:16:00.065+0000 7fdd34f51640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 40012) UID: 1000 2024-04-06T08:16:01.259 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:16:01.256+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d90acf70 2024-04-06T08:16:05.068 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T08:16:05.068 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:16:05.069 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T08:16:05.069 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T08:16:05.069 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T08:16:05.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T08:16:05.072 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-04-06T08:16:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-06T08:16:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'image health:' 2024-04-06T08:16:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 3 2024-04-06T08:16:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=OK 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test OK = ERROR 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test OK = OK 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_count=50 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local count 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_count=0 2024-04-06T08:16:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 40 2024-04-06T08:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 1 -ne 1 2024-04-06T08:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T08:16:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'images: ' 2024-04-06T08:16:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 2 2024-04-06T08:16:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ count=50 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 50 = 50 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state 2024-04-06T08:16:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 40 2024-04-06T08:16:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:16:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 1 -ne 1 2024-04-06T08:16:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T08:16:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'image health:' 2024-04-06T08:16:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 3 2024-04-06T08:16:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:16:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=WARNING 2024-04-06T08:16:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test WARNING = ERROR 2024-04-06T08:16:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test WARNING = OK 2024-04-06T08:16:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:16:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 2 -ne 1 2024-04-06T08:16:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 30 2024-04-06T08:16:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T08:16:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'image health:' 2024-04-06T08:16:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 3 2024-04-06T08:16:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:16:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=OK 2024-04-06T08:16:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test OK = ERROR 2024-04-06T08:16:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test OK = OK 2024-04-06T08:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: compare many images' 2024-04-06T08:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T08:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T08:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 08:16:47 TEST: compare many images 2024-04-06T08:16:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 08:16:47 TEST: compare many images 2024-04-06T08:16:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 50 2024-04-06T08:16:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:16:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_1 2024-04-06T08:16:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_1 snap 2024-04-06T08:16:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:16:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:16:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:16:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:16:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:16:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:16:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-04-06T08:16:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:16:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:16:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:16:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:16:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:16:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:16:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:16:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:16:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:16:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:16:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-04-06T08:16:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:16:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:16:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:16:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:16:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:16:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_1.status 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_1.status 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:16:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.status 2024-04-06T08:16:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_1.status 2024-04-06T08:16:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_1.status 2024-04-06T08:16:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:16:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-04-06T08:16:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-04-06T08:16:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 119 2024-04-06T08:16:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.status 2024-04-06T08:16:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.status 2024-04-06T08:16:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-04-06T08:16:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:16:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=119 2024-04-06T08:16:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:16:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:16:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:16:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-06T08:16:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:16:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=68 2024-04-06T08:16:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 119 = 68 2024-04-06T08:16:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:16:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:16:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-06T08:16:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:16:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:16:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:16:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:16:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 119 = '' 2024-04-06T08:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:52.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-06T08:16:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:16:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:16:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:16:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:16:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:16:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:16:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 119 = '' 2024-04-06T08:16:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:16:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.status 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-06T08:16:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:16:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=119 2024-04-06T08:16:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 119 = 119 2024-04-06T08:16:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-04-06T08:16:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:16:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:16:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:16:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:16:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_1': 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10ad1994a4cc 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10ad1994a4cc 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:06:54 2024 2024-04-06T08:16:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:06:54 2024 2024-04-06T08:16:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:06:54 2024 2024-04-06T08:16:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:16:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:16:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:16:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: eb012d1c-4f5e-44aa-9edc-096c064d4f16 2024-04-06T08:16:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:16:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_1 snap 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.export 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.export 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.export 2024-04-06T08:16:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.export 2024-04-06T08:16:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:16:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.export 2024-04-06T08:16:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:16:55.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8718484655170579) > weight (0.25) 2024-04-06T08:16:55.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6374516510963203) > weight (0.25) 2024-04-06T08:16:55.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.42428629405329865) > weight (0.25) 2024-04-06T08:16:55.967 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-06T08:16:55.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-06T08:16:55.968 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T08:16:55.968 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-06T08:16:55.968 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:16:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.export 2024-04-06T08:16:55.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:16:55.964+0000 7fcfd126d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 42100) UID: 1000 2024-04-06T08:16:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_1.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_1.export 2024-04-06T08:16:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:16:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_2 2024-04-06T08:16:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_2 snap 2024-04-06T08:16:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:16:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:16:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:16:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:16:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:16:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-04-06T08:16:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:16:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:16:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:16:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:16:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:16:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:16:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:16:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:16:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:16:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:16:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:16:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-04-06T08:16:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:16:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:16:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:16:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:16:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:16:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:16:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:16:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-04-06T08:16:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-06T08:16:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:16:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:16:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:16:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:16:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:16:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:16:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:16:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:16:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:16:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:16:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:16:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:16:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:16:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:16:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-04-06T08:16:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:no valid command found; 1 closest matches: 2024-04-06T08:16:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status mirror/image_26 2024-04-06T08:16:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-04-06T08:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:16:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:16:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:16:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:16:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:16:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:16:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:16:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:16:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-04-06T08:16:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:16:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:16:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:16:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:16:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:16:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_2.status 2024-04-06T08:16:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_2.status 2024-04-06T08:16:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:16:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.status 2024-04-06T08:16:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_2.status 2024-04-06T08:16:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_2.status 2024-04-06T08:16:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:16:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:16:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-04-06T08:16:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-04-06T08:16:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 121 2024-04-06T08:16:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.status 2024-04-06T08:16:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:16:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.status 2024-04-06T08:16:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-04-06T08:16:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:16:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=121 2024-04-06T08:16:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:16:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:16:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:16:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:16:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:16:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:16:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:16:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:16:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-06T08:16:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:16:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:16:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:16:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:16:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:16:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:16:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:16:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 121 = '' 2024-04-06T08:16:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:16:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:17:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:17:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:17:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-06T08:17:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 121 = '' 2024-04-06T08:17:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:17:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:17:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.status 2024-04-06T08:17:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-06T08:17:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=121 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 121 = 121 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:02.070 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-06T08:17:02.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-06T08:17:02.070 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-06T08:17:02.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-06T08:17:02.074 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-04-06T08:17:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_2': 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de7a55e84a 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de7a55e84a 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:10 2024 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:10 2024 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:10 2024 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 594aad0b-58bb-4181-a369-5cf04943564e 2024-04-06T08:17:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_2 snap 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.export 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.export 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.export 2024-04-06T08:17:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.export 2024-04-06T08:17:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.export 2024-04-06T08:17:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.export 2024-04-06T08:17:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_2.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_2.export 2024-04-06T08:17:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_3 2024-04-06T08:17:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_3 snap 2024-04-06T08:17:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-04-06T08:17:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-04-06T08:17:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-04-06T08:17:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_3.status 2024-04-06T08:17:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_3.status 2024-04-06T08:17:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.status 2024-04-06T08:17:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_3.status 2024-04-06T08:17:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_3.status 2024-04-06T08:17:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-04-06T08:17:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-04-06T08:17:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 123 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.status 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.status 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-04-06T08:17:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=123 2024-04-06T08:17:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-06T08:17:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=70 2024-04-06T08:17:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 123 = 70 2024-04-06T08:17:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-06T08:17:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 123 = '' 2024-04-06T08:17:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-06T08:17:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 123 = '' 2024-04-06T08:17:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:17:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.status 2024-04-06T08:17:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-06T08:17:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=123 2024-04-06T08:17:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 123 = 123 2024-04-06T08:17:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-04-06T08:17:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_3': 2024-04-06T08:17:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 117cef11a8ed 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.117cef11a8ed 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:15 2024 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:15 2024 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:15 2024 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 23b023dd-adbb-4aae-bbd9-c320028b0492 2024-04-06T08:17:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_3 snap 2024-04-06T08:17:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:17:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.export 2024-04-06T08:17:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.export 2024-04-06T08:17:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.export 2024-04-06T08:17:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.export 2024-04-06T08:17:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.export 2024-04-06T08:17:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.export 2024-04-06T08:17:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_3.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_3.export 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_4 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_4 snap 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-04-06T08:17:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-04-06T08:17:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-04-06T08:17:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:no valid command found; 3 closest matches: 2024-04-06T08:17:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status mirror/image_42 2024-04-06T08:17:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status mirror/image_44 2024-04-06T08:17:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status mirror/image_48 2024-04-06T08:17:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:17:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:17:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:17:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:17:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:17:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-04-06T08:17:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:no valid command found; 2 closest matches: 2024-04-06T08:17:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status mirror/image_40 2024-04-06T08:17:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status mirror/image_46 2024-04-06T08:17:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:17:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-04-06T08:17:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:17:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:17:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:17:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:17:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:17:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:17:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-04-06T08:17:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_4.status 2024-04-06T08:17:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_4.status 2024-04-06T08:17:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.status 2024-04-06T08:17:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_4.status 2024-04-06T08:17:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_4.status 2024-04-06T08:17:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-04-06T08:17:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-04-06T08:17:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 125 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.status 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.status 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-04-06T08:17:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=125 2024-04-06T08:17:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-06T08:17:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 125 = '' 2024-04-06T08:17:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-06T08:17:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 125 = '' 2024-04-06T08:17:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.status 2024-04-06T08:17:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-06T08:17:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=125 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 125 = 125 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-04-06T08:17:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_4': 2024-04-06T08:17:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12412559c004 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12412559c004 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:26 2024 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:26 2024 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:26 2024 2024-04-06T08:17:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ca09eaed-79f4-4ee1-8861-740c13842056 2024-04-06T08:17:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_4 snap 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.export 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.export 2024-04-06T08:17:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.export 2024-04-06T08:17:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.export 2024-04-06T08:17:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.export 2024-04-06T08:17:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.export 2024-04-06T08:17:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_4.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_4.export 2024-04-06T08:17:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_5 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_5 snap 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-04-06T08:17:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:22.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-04-06T08:17:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-04-06T08:17:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-04-06T08:17:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-04-06T08:17:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_5.status 2024-04-06T08:17:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_5.status 2024-04-06T08:17:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.status 2024-04-06T08:17:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_5.status 2024-04-06T08:17:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_5.status 2024-04-06T08:17:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-04-06T08:17:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-04-06T08:17:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 127 2024-04-06T08:17:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.status 2024-04-06T08:17:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.status 2024-04-06T08:17:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-04-06T08:17:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=127 2024-04-06T08:17:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-06T08:17:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=72 2024-04-06T08:17:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 127 = 72 2024-04-06T08:17:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-06T08:17:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 127 = '' 2024-04-06T08:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:25.073 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5765542339410712) > weight (0.25) 2024-04-06T08:17:25.073 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4225347165973493) > weight (0.25) 2024-04-06T08:17:25.073 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6781104964977771) > weight (0.25) 2024-04-06T08:17:25.073 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-06T08:17:25.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-06T08:17:25.074 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-06T08:17:25.074 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-06T08:17:25.074 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:17:25.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:17:25.071+0000 7fb644bd3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 35008) UID: 1000 2024-04-06T08:17:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-06T08:17:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 127 = '' 2024-04-06T08:17:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:17:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.status 2024-04-06T08:17:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-06T08:17:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=127 2024-04-06T08:17:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 127 = 127 2024-04-06T08:17:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_5': 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1286a2567724 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1286a2567724 2024-04-06T08:17:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:41 2024 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:41 2024 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:41 2024 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: b950a79f-80c3-4891-a04f-2ec8f28298d1 2024-04-06T08:17:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_5 snap 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.export 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.export 2024-04-06T08:17:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.export 2024-04-06T08:17:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.export 2024-04-06T08:17:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.export 2024-04-06T08:17:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.export 2024-04-06T08:17:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_5.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_5.export 2024-04-06T08:17:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_6 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_6 snap 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:31.176 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-06T08:17:31.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-06T08:17:31.176 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-06T08:17:31.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-06T08:17:31.179 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-04-06T08:17:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-04-06T08:17:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-04-06T08:17:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_6.status 2024-04-06T08:17:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_6.status 2024-04-06T08:17:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.status 2024-04-06T08:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_6.status 2024-04-06T08:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_6.status 2024-04-06T08:17:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-04-06T08:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-04-06T08:17:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 129 2024-04-06T08:17:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.status 2024-04-06T08:17:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.status 2024-04-06T08:17:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-04-06T08:17:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=129 2024-04-06T08:17:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-06T08:17:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=73 2024-04-06T08:17:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 129 = 73 2024-04-06T08:17:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-06T08:17:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 129 = '' 2024-04-06T08:17:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-06T08:17:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 129 = '' 2024-04-06T08:17:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:17:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.status 2024-04-06T08:17:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-06T08:17:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=129 2024-04-06T08:17:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 129 = 129 2024-04-06T08:17:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-04-06T08:17:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_6': 2024-04-06T08:17:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 117ce4766496 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.117ce4766496 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:07:54 2024 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:07:54 2024 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:07:54 2024 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 6347dabe-e24c-40dc-91f2-8040b6675422 2024-04-06T08:17:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_6 snap 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.export 2024-04-06T08:17:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.export 2024-04-06T08:17:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.export 2024-04-06T08:17:37.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.export 2024-04-06T08:17:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.export 2024-04-06T08:17:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.export 2024-04-06T08:17:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_6.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_6.export 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_7 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_7 snap 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-04-06T08:17:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-04-06T08:17:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-04-06T08:17:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-06T08:17:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:17:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:17:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-04-06T08:17:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:17:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:17:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:17:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:17:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:17:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:17:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:17:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-04-06T08:17:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_7.status 2024-04-06T08:17:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_7.status 2024-04-06T08:17:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.status 2024-04-06T08:17:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_7.status 2024-04-06T08:17:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_7.status 2024-04-06T08:17:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-04-06T08:17:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-04-06T08:17:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 131 2024-04-06T08:17:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.status 2024-04-06T08:17:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.status 2024-04-06T08:17:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-04-06T08:17:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=131 2024-04-06T08:17:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-06T08:17:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=74 2024-04-06T08:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 131 = 74 2024-04-06T08:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-06T08:17:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 131 = '' 2024-04-06T08:17:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-06T08:17:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 131 = '' 2024-04-06T08:17:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:17:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.status 2024-04-06T08:17:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-06T08:17:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=131 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 131 = 131 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-04-06T08:17:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_7': 2024-04-06T08:17:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1241ab984bd1 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1241ab984bd1 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:08 2024 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:08 2024 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:08 2024 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 839a8373-291e-4850-b111-46075edd9765 2024-04-06T08:17:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_7 snap 2024-04-06T08:17:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:17:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.export 2024-04-06T08:17:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.export 2024-04-06T08:17:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.export 2024-04-06T08:17:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.export 2024-04-06T08:17:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.export 2024-04-06T08:17:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.export 2024-04-06T08:17:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_7.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_7.export 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_8 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_8 snap 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-04-06T08:17:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-04-06T08:17:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-06T08:17:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:17:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:17:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-04-06T08:17:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:17:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:17:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:17:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:17:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-04-06T08:17:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-06T08:17:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:17:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:17:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-04-06T08:17:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:17:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:17:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:17:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:17:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:17:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:17:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:17:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:17:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:17:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:17:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:17:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-04-06T08:17:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_8.status 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_8.status 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.status 2024-04-06T08:17:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_8.status 2024-04-06T08:17:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_8.status 2024-04-06T08:17:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-04-06T08:17:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-04-06T08:17:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 133 2024-04-06T08:17:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.status 2024-04-06T08:17:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.status 2024-04-06T08:17:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-04-06T08:17:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=133 2024-04-06T08:17:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-06T08:17:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=75 2024-04-06T08:17:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 133 = 75 2024-04-06T08:17:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:17:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-06T08:17:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 133 = '' 2024-04-06T08:17:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:17:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-06T08:17:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 133 = '' 2024-04-06T08:17:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:17:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.status 2024-04-06T08:17:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-06T08:17:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=133 2024-04-06T08:17:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 133 = 133 2024-04-06T08:17:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:17:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-04-06T08:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_8': 2024-04-06T08:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:17:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1286221fbe15 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1286221fbe15 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:18 2024 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:18 2024 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:18 2024 2024-04-06T08:17:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: be4f8eb9-e09c-4224-856e-896364fa79d2 2024-04-06T08:17:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:17:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_8 snap 2024-04-06T08:17:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:17:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:17:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.export 2024-04-06T08:17:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.export 2024-04-06T08:17:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.export 2024-04-06T08:17:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.export 2024-04-06T08:17:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.export 2024-04-06T08:17:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:17:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.export 2024-04-06T08:17:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_8.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_8.export 2024-04-06T08:17:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_9 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_9 snap 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:17:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:17:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-04-06T08:17:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:17:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-04-06T08:17:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:17:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:17:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:17:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:17:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:17:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-04-06T08:17:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:17:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:17:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:17:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:17:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:17:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:17:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:17:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:17:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-04-06T08:17:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_9.status 2024-04-06T08:17:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_9.status 2024-04-06T08:17:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.status 2024-04-06T08:17:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_9.status 2024-04-06T08:17:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_9.status 2024-04-06T08:17:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:17:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:17:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-04-06T08:17:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-04-06T08:17:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 135 2024-04-06T08:17:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.status 2024-04-06T08:17:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:17:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.status 2024-04-06T08:17:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-04-06T08:17:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:17:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=135 2024-04-06T08:17:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-06T08:17:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:17:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:17:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:17:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:17:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:17:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:17:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:17:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 135 = '' 2024-04-06T08:17:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:17:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:18:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:18:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:18:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-06T08:18:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 135 = '' 2024-04-06T08:18:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:18:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:18:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.status 2024-04-06T08:18:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-06T08:18:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=135 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 135 = 135 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:18:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-04-06T08:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_9': 2024-04-06T08:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de3fa66694 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de3fa66694 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:30 2024 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:30 2024 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:30 2024 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 760529a0-2277-4e6f-aa85-bf596b26afe0 2024-04-06T08:18:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_9 snap 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.export 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.export 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.export 2024-04-06T08:18:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.export 2024-04-06T08:18:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.export 2024-04-06T08:18:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.export 2024-04-06T08:18:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_9.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_9.export 2024-04-06T08:18:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_10 2024-04-06T08:18:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_10 snap 2024-04-06T08:18:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-04-06T08:18:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:18:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:18:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:18:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:18:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:18:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:18:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:18:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:18:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:18:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:18:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:18:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-04-06T08:18:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:18:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:18:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:18:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:18:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:18:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:18:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:18:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_10.status 2024-04-06T08:18:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_10.status 2024-04-06T08:18:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.status 2024-04-06T08:18:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_10.status 2024-04-06T08:18:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_10.status 2024-04-06T08:18:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-04-06T08:18:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-04-06T08:18:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 137 2024-04-06T08:18:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.status 2024-04-06T08:18:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.status 2024-04-06T08:18:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-04-06T08:18:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:18:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=137 2024-04-06T08:18:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:18:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-06T08:18:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 137 = '' 2024-04-06T08:18:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-06T08:18:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 137 = '' 2024-04-06T08:18:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.status 2024-04-06T08:18:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-06T08:18:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=137 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 137 = 137 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-04-06T08:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_10': 2024-04-06T08:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128f2602c671 2024-04-06T08:18:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128f2602c671 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:43 2024 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:43 2024 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:43 2024 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 1626ab18-b923-4d9b-b871-b9490fbd81d7 2024-04-06T08:18:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_10 snap 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.export 2024-04-06T08:18:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.export 2024-04-06T08:18:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.export 2024-04-06T08:18:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.export 2024-04-06T08:18:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.export 2024-04-06T08:18:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.export 2024-04-06T08:18:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_10.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_10.export 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_11 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_11 snap 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-04-06T08:18:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-04-06T08:18:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:18:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:18:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:18:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-04-06T08:18:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:18:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:18:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:18:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:18:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:18:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:18:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:18:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:18:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:18:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:18:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-04-06T08:18:13.075 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8004718105227) > weight (0.25) 2024-04-06T08:18:13.075 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7103461806071786) > weight (0.25) 2024-04-06T08:18:13.075 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T08:18:13.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T08:18:13.075 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-06T08:18:13.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:18:13.070+0000 7fe61bef4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 30451) UID: 1000 2024-04-06T08:18:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-06T08:18:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:18:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:18:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-04-06T08:18:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:18:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-04-06T08:18:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:18:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:18:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:18:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:18:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_11.status 2024-04-06T08:18:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_11.status 2024-04-06T08:18:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.status 2024-04-06T08:18:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_11.status 2024-04-06T08:18:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_11.status 2024-04-06T08:18:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-04-06T08:18:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-04-06T08:18:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 139 2024-04-06T08:18:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.status 2024-04-06T08:18:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.status 2024-04-06T08:18:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:18:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=139 2024-04-06T08:18:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:18:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=78 2024-04-06T08:18:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 139 = 78 2024-04-06T08:18:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 139 = '' 2024-04-06T08:18:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 139 = '' 2024-04-06T08:18:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:18:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 139 = '' 2024-04-06T08:18:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 2 2024-04-06T08:18:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 139 = '' 2024-04-06T08:18:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 2 2024-04-06T08:18:20.076 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T08:18:20.077 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:18:20.077 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T08:18:20.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T08:18:20.078 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T08:18:20.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T08:18:20.081 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-04-06T08:18:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.status 2024-04-06T08:18:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-06T08:18:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=139 2024-04-06T08:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 139 = 139 2024-04-06T08:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-04-06T08:18:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_11': 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128f3eb7f44c 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128f3eb7f44c 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:08:50 2024 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:08:50 2024 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:08:50 2024 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f7102142-b3e4-44ed-9508-3bf8555f6258 2024-04-06T08:18:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_11 snap 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.export 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.export 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.export 2024-04-06T08:18:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.export 2024-04-06T08:18:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.export 2024-04-06T08:18:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.export 2024-04-06T08:18:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_11.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_11.export 2024-04-06T08:18:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_12 2024-04-06T08:18:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_12 snap 2024-04-06T08:18:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-04-06T08:18:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:18:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:18:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:18:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:18:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:18:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:18:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-04-06T08:18:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:18:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:18:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:18:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:18:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:18:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:18:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:18:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-04-06T08:18:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:18:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:18:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:18:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:18:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:18:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:18:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:18:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_12.status 2024-04-06T08:18:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_12.status 2024-04-06T08:18:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.status 2024-04-06T08:18:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_12.status 2024-04-06T08:18:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_12.status 2024-04-06T08:18:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-04-06T08:18:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-04-06T08:18:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 141 2024-04-06T08:18:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.status 2024-04-06T08:18:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.status 2024-04-06T08:18:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-04-06T08:18:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:18:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=141 2024-04-06T08:18:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:18:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-06T08:18:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=79 2024-04-06T08:18:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 141 = 79 2024-04-06T08:18:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-06T08:18:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 141 = '' 2024-04-06T08:18:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-06T08:18:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 141 = '' 2024-04-06T08:18:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:18:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.status 2024-04-06T08:18:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-06T08:18:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=141 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 141 = 141 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-04-06T08:18:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_12': 2024-04-06T08:18:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128f32d472c3 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128f32d472c3 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:02 2024 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:02 2024 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:02 2024 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8901ee92-51d2-43ee-985f-683952d0fb7e 2024-04-06T08:18:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_12 snap 2024-04-06T08:18:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:18:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.export 2024-04-06T08:18:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.export 2024-04-06T08:18:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.export 2024-04-06T08:18:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.export 2024-04-06T08:18:31.568 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:18:31.566+0000 7f5a15134640 -1 reset not still connected to 0x55db9bc24410 2024-04-06T08:18:31.568 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:18:31.566+0000 7f5a15134640 -1 reset not still connected to 0x55dbb44f88f0 2024-04-06T08:18:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.export 2024-04-06T08:18:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.export 2024-04-06T08:18:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_12.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_12.export 2024-04-06T08:18:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_13 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_13 snap 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-04-06T08:18:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-04-06T08:18:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:18:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:18:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:18:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:18:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:18:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:18:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:18:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-04-06T08:18:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:18:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:18:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:18:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:18:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:18:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:18:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:18:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-04-06T08:18:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:18:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:18:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-04-06T08:18:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_13.status 2024-04-06T08:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_13.status 2024-04-06T08:18:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.status 2024-04-06T08:18:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_13.status 2024-04-06T08:18:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_13.status 2024-04-06T08:18:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-04-06T08:18:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-04-06T08:18:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 143 2024-04-06T08:18:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.status 2024-04-06T08:18:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.status 2024-04-06T08:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-04-06T08:18:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:18:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=143 2024-04-06T08:18:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:18:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-06T08:18:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=80 2024-04-06T08:18:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 143 = 80 2024-04-06T08:18:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-06T08:18:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=80 2024-04-06T08:18:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 143 = 80 2024-04-06T08:18:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-06T08:18:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 143 = '' 2024-04-06T08:18:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:18:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.status 2024-04-06T08:18:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-06T08:18:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=143 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 143 = 143 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-04-06T08:18:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_13': 2024-04-06T08:18:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10ded0d697da 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10ded0d697da 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:11 2024 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:11 2024 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:11 2024 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 580a3f05-2cbb-49eb-bcd9-418388fdf451 2024-04-06T08:18:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_13 snap 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.export 2024-04-06T08:18:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.export 2024-04-06T08:18:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.export 2024-04-06T08:18:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.export 2024-04-06T08:18:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.export 2024-04-06T08:18:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.export 2024-04-06T08:18:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_13.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_13.export 2024-04-06T08:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_14 2024-04-06T08:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_14 snap 2024-04-06T08:18:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:18:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:18:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:18:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:18:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:18:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-04-06T08:18:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:18:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:18:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:18:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:18:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:18:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-04-06T08:18:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T08:18:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:18:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:18:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-04-06T08:18:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:18:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:18:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:18:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:18:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-04-06T08:18:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T08:18:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:18:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:18:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:18:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:18:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:18:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:18:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:18:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:18:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:18:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-04-06T08:18:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-06T08:18:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:18:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:18:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:18:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:18:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:18:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:18:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:18:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:18:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:18:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-04-06T08:18:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:18:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:18:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:18:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:18:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_14.status 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_14.status 2024-04-06T08:18:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.status 2024-04-06T08:18:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_14.status 2024-04-06T08:18:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_14.status 2024-04-06T08:18:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-04-06T08:18:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-04-06T08:18:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 145 2024-04-06T08:18:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.status 2024-04-06T08:18:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.status 2024-04-06T08:18:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-04-06T08:18:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:18:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=145 2024-04-06T08:18:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:18:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-06T08:18:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 145 = '' 2024-04-06T08:18:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-06T08:18:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 145 = '' 2024-04-06T08:18:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.status 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-06T08:18:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=145 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 145 = 145 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-04-06T08:18:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_14': 2024-04-06T08:18:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1241e879356e 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1241e879356e 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:19 2024 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:19 2024 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:19 2024 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: c583fe36-b4d5-459c-9629-9b6a5c24b1c2 2024-04-06T08:18:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_14 snap 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.export 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.export 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.export 2024-04-06T08:18:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.export 2024-04-06T08:18:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.export 2024-04-06T08:18:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.export 2024-04-06T08:18:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_14.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_14.export 2024-04-06T08:18:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_15 2024-04-06T08:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_15 snap 2024-04-06T08:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-04-06T08:18:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:18:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:18:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:18:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:18:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:18:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-04-06T08:18:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:18:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:18:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:18:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:18:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:18:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:18:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-04-06T08:18:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T08:18:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:18:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:18:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:18:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:18:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:18:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:18:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:18:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:18:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-04-06T08:18:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-06T08:18:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:18:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:18:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:18:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-04-06T08:18:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:18:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:18:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:18:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:18:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:18:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:18:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:18:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:18:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-04-06T08:18:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:18:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:18:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:18:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:18:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:18:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_15.status 2024-04-06T08:18:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_15.status 2024-04-06T08:18:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.status 2024-04-06T08:18:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_15.status 2024-04-06T08:18:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_15.status 2024-04-06T08:18:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:18:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-04-06T08:18:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-04-06T08:18:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 147 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.status 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.status 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-04-06T08:18:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:18:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=147 2024-04-06T08:18:53.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:18:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-06T08:18:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=82 2024-04-06T08:18:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 147 = 82 2024-04-06T08:18:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:18:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-06T08:18:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 147 = '' 2024-04-06T08:18:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:18:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-06T08:18:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:18:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:18:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:18:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:18:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:18:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 147 = '' 2024-04-06T08:18:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:18:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:18:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.status 2024-04-06T08:18:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-06T08:18:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:18:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=147 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 147 = 147 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:18:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:18:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-04-06T08:18:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_15': 2024-04-06T08:18:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1286e8573069 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1286e8573069 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:28 2024 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:28 2024 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:28 2024 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 9ab9634b-f0c0-42f0-857e-b5d198938167 2024-04-06T08:18:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_15 snap 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.export 2024-04-06T08:18:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.export 2024-04-06T08:18:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.export 2024-04-06T08:18:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.export 2024-04-06T08:18:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.export 2024-04-06T08:18:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:18:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.export 2024-04-06T08:18:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_15.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_15.export 2024-04-06T08:18:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:18:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:18:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_16 2024-04-06T08:18:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_16 snap 2024-04-06T08:18:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:18:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:18:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:18:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:18:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-04-06T08:18:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:18:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:18:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-04-06T08:19:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-06T08:19:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T08:19:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-06T08:19:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T08:19:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-04-06T08:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:19:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:19:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:19:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-04-06T08:19:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-06T08:19:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:19:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:19:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-04-06T08:19:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_16.status 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_16.status 2024-04-06T08:19:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.status 2024-04-06T08:19:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_16.status 2024-04-06T08:19:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_16.status 2024-04-06T08:19:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-04-06T08:19:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-04-06T08:19:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 149 2024-04-06T08:19:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.status 2024-04-06T08:19:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.status 2024-04-06T08:19:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-04-06T08:19:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=149 2024-04-06T08:19:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-06T08:19:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=83 2024-04-06T08:19:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 149 = 83 2024-04-06T08:19:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-06T08:19:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 149 = '' 2024-04-06T08:19:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-06T08:19:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 149 = '' 2024-04-06T08:19:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.status 2024-04-06T08:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-06T08:19:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=149 2024-04-06T08:19:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 149 = 149 2024-04-06T08:19:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-04-06T08:19:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-04-06T08:19:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_16': 2024-04-06T08:19:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128f8bedb57f 2024-04-06T08:19:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128f8bedb57f 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:36 2024 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:36 2024 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:36 2024 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 725496c1-43b8-4408-a32f-b66992d7c9ef 2024-04-06T08:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_16 snap 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.export 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.export 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.export 2024-04-06T08:19:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.export 2024-04-06T08:19:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.export 2024-04-06T08:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.export 2024-04-06T08:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_16.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_16.export 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_17 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_17 snap 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-04-06T08:19:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-04-06T08:19:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-04-06T08:19:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-06T08:19:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-04-06T08:19:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-04-06T08:19:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_17.status 2024-04-06T08:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_17.status 2024-04-06T08:19:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.status 2024-04-06T08:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_17.status 2024-04-06T08:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_17.status 2024-04-06T08:19:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-04-06T08:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-04-06T08:19:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 151 2024-04-06T08:19:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.status 2024-04-06T08:19:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.status 2024-04-06T08:19:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-04-06T08:19:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=151 2024-04-06T08:19:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-06T08:19:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=84 2024-04-06T08:19:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 151 = 84 2024-04-06T08:19:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-06T08:19:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 151 = '' 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-06T08:19:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 151 = '' 2024-04-06T08:19:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:19:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.status 2024-04-06T08:19:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-06T08:19:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=151 2024-04-06T08:19:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 151 = 151 2024-04-06T08:19:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-04-06T08:19:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_17': 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10ded5d8bf10 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10ded5d8bf10 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:42 2024 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:42 2024 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:42 2024 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ffce73d4-43fe-46e8-9362-ed0490dd6ab0 2024-04-06T08:19:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_17 snap 2024-04-06T08:19:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.export 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.export 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.export 2024-04-06T08:19:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.export 2024-04-06T08:19:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.export 2024-04-06T08:19:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.export 2024-04-06T08:19:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_17.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_17.export 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_18 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_18 snap 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-04-06T08:19:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-04-06T08:19:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T08:19:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-04-06T08:19:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:18.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:18.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:18.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-04-06T08:19:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-06T08:19:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-04-06T08:19:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:19:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:19:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:19:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:19:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-04-06T08:19:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_18.status 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_18.status 2024-04-06T08:19:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.status 2024-04-06T08:19:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_18.status 2024-04-06T08:19:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_18.status 2024-04-06T08:19:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-04-06T08:19:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-04-06T08:19:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 153 2024-04-06T08:19:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.status 2024-04-06T08:19:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.status 2024-04-06T08:19:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-04-06T08:19:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=153 2024-04-06T08:19:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-06T08:19:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 153 = '' 2024-04-06T08:19:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-06T08:19:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 153 = '' 2024-04-06T08:19:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.status 2024-04-06T08:19:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-06T08:19:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=153 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 153 = 153 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-04-06T08:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_18': 2024-04-06T08:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 124166c03f49 2024-04-06T08:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.124166c03f49 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:09:56 2024 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:09:56 2024 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:09:56 2024 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 2a2e0364-70be-4a9d-8f86-d8dd2c9b39fc 2024-04-06T08:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_18 snap 2024-04-06T08:19:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:19:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.export 2024-04-06T08:19:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.export 2024-04-06T08:19:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.export 2024-04-06T08:19:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.export 2024-04-06T08:19:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.export 2024-04-06T08:19:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.export 2024-04-06T08:19:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_18.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_18.export 2024-04-06T08:19:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_19 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_19 snap 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:25.180 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T08:19:25.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T08:19:25.181 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T08:19:25.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:19:25.181+0000 7fc4845d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40723) UID: 1000 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-04-06T08:19:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-04-06T08:19:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-04-06T08:19:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-06T08:19:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-04-06T08:19:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_19.status 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_19.status 2024-04-06T08:19:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.status 2024-04-06T08:19:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_19.status 2024-04-06T08:19:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_19.status 2024-04-06T08:19:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-04-06T08:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-04-06T08:19:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 155 2024-04-06T08:19:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.status 2024-04-06T08:19:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.status 2024-04-06T08:19:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-04-06T08:19:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=155 2024-04-06T08:19:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-06T08:19:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=86 2024-04-06T08:19:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 155 = 86 2024-04-06T08:19:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-06T08:19:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 155 = '' 2024-04-06T08:19:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:28.182 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T08:19:28.182 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:19:28.183 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T08:19:28.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T08:19:28.183 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T08:19:28.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T08:19:28.186 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-04-06T08:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-06T08:19:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 155 = '' 2024-04-06T08:19:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:19:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.status 2024-04-06T08:19:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-06T08:19:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=155 2024-04-06T08:19:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 155 = 155 2024-04-06T08:19:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-04-06T08:19:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-04-06T08:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_19': 2024-04-06T08:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1286a2af615f 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1286a2af615f 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:05 2024 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:05 2024 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:05 2024 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 73a422d7-74ad-4197-90fc-2e353882f67e 2024-04-06T08:19:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_19 snap 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.export 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.export 2024-04-06T08:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.export 2024-04-06T08:19:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.export 2024-04-06T08:19:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.export 2024-04-06T08:19:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.export 2024-04-06T08:19:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_19.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_19.export 2024-04-06T08:19:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_20 2024-04-06T08:19:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_20 snap 2024-04-06T08:19:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-04-06T08:19:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T08:19:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-04-06T08:19:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T08:19:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:19:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:19:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:35.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-04-06T08:19:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-06T08:19:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-04-06T08:19:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:19:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:19:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:19:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:19:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:19:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-04-06T08:19:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_20.status 2024-04-06T08:19:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_20.status 2024-04-06T08:19:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.status 2024-04-06T08:19:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_20.status 2024-04-06T08:19:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_20.status 2024-04-06T08:19:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-04-06T08:19:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-04-06T08:19:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 157 2024-04-06T08:19:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.status 2024-04-06T08:19:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.status 2024-04-06T08:19:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-04-06T08:19:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=157 2024-04-06T08:19:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-06T08:19:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 157 = '' 2024-04-06T08:19:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-06T08:19:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 157 = '' 2024-04-06T08:19:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.status 2024-04-06T08:19:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-06T08:19:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=157 2024-04-06T08:19:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 157 = 157 2024-04-06T08:19:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-04-06T08:19:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_20': 2024-04-06T08:19:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128fc98ed0bd 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128fc98ed0bd 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:18 2024 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:18 2024 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:18 2024 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 877e6897-26c6-469a-b566-eb8cf289549d 2024-04-06T08:19:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_20 snap 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.export 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.export 2024-04-06T08:19:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.export 2024-04-06T08:19:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.export 2024-04-06T08:19:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.export 2024-04-06T08:19:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.export 2024-04-06T08:19:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_20.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_20.export 2024-04-06T08:19:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_21 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_21 snap 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:41.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-04-06T08:19:41.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-04-06T08:19:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-04-06T08:19:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-06T08:19:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-04-06T08:19:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_21.status 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_21.status 2024-04-06T08:19:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.status 2024-04-06T08:19:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_21.status 2024-04-06T08:19:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_21.status 2024-04-06T08:19:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-04-06T08:19:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-04-06T08:19:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 159 2024-04-06T08:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.status 2024-04-06T08:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.status 2024-04-06T08:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-04-06T08:19:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=159 2024-04-06T08:19:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-06T08:19:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=88 2024-04-06T08:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 159 = 88 2024-04-06T08:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-06T08:19:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=88 2024-04-06T08:19:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 159 = 88 2024-04-06T08:19:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-06T08:19:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 159 = '' 2024-04-06T08:19:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:19:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.status 2024-04-06T08:19:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-06T08:19:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=159 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 159 = 159 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-04-06T08:19:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_21': 2024-04-06T08:19:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10dec0d0c533 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10dec0d0c533 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:23 2024 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:23 2024 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:23 2024 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: a30e9377-aa3c-4d27-9702-2aab3727fd99 2024-04-06T08:19:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_21 snap 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.export 2024-04-06T08:19:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.export 2024-04-06T08:19:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.export 2024-04-06T08:19:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.export 2024-04-06T08:19:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.export 2024-04-06T08:19:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.export 2024-04-06T08:19:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_21.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_21.export 2024-04-06T08:19:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_22 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_22 snap 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-04-06T08:19:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:19:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:19:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:19:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-04-06T08:19:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:19:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:19:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:19:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:19:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:19:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:19:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:19:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:19:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:19:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:19:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:19:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-04-06T08:19:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T08:19:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:19:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:19:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:19:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:19:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:19:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-04-06T08:19:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T08:19:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:19:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:19:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:19:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:19:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:19:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:19:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:19:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-04-06T08:19:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-06T08:19:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:19:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:19:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:19:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-04-06T08:19:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:19:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:19:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:19:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:19:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:19:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:19:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:19:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:19:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-04-06T08:19:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:19:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:19:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:19:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:19:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:19:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:19:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:19:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_22.status 2024-04-06T08:19:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_22.status 2024-04-06T08:19:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.status 2024-04-06T08:19:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_22.status 2024-04-06T08:19:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_22.status 2024-04-06T08:19:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:19:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-04-06T08:19:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-04-06T08:19:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 161 2024-04-06T08:19:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.status 2024-04-06T08:19:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.status 2024-04-06T08:19:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-04-06T08:19:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=161 2024-04-06T08:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-06T08:19:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=89 2024-04-06T08:19:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 161 = 89 2024-04-06T08:19:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:19:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-06T08:19:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=89 2024-04-06T08:19:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 161 = 89 2024-04-06T08:19:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:19:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-06T08:19:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:19:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:19:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:19:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 161 = '' 2024-04-06T08:19:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:19:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:19:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.status 2024-04-06T08:19:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-06T08:19:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:19:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=161 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 161 = 161 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:19:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:19:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-04-06T08:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_22': 2024-04-06T08:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:19:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1298b1e563f6 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1298b1e563f6 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:31 2024 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:31 2024 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:31 2024 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ac3a5da1-f39e-465e-a835-c87e8aa02575 2024-04-06T08:19:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:19:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_22 snap 2024-04-06T08:19:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:19:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:19:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.export 2024-04-06T08:19:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.export 2024-04-06T08:19:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.export 2024-04-06T08:19:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.export 2024-04-06T08:19:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.export 2024-04-06T08:19:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:19:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.export 2024-04-06T08:19:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_22.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_22.export 2024-04-06T08:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_23 2024-04-06T08:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_23 snap 2024-04-06T08:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:19:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:19:59.081 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3214821218302546) > weight (0.25) 2024-04-06T08:19:59.082 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-06T08:19:59.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-06T08:19:59.082 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T08:19:59.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:19:59.080+0000 7f954153d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 37352) UID: 1000 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:19:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:19:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:19:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:19:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:19:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:19:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-04-06T08:20:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-06T08:20:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-04-06T08:20:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-04-06T08:20:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_23.status 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_23.status 2024-04-06T08:20:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.status 2024-04-06T08:20:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_23.status 2024-04-06T08:20:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_23.status 2024-04-06T08:20:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-04-06T08:20:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-04-06T08:20:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 163 2024-04-06T08:20:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.status 2024-04-06T08:20:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.status 2024-04-06T08:20:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-04-06T08:20:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=163 2024-04-06T08:20:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:02.083 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-06T08:20:02.083 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:20:02.083 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-06T08:20:02.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T08:20:02.083 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T08:20:02.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T08:20:02.086 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-06T08:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=90 2024-04-06T08:20:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 163 = 90 2024-04-06T08:20:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-06T08:20:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 163 = '' 2024-04-06T08:20:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-06T08:20:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 163 = '' 2024-04-06T08:20:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:20:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:05.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.status 2024-04-06T08:20:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-06T08:20:05.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=163 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 163 = 163 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_23': 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 129882533b33 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.129882533b33 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:44 2024 2024-04-06T08:20:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:44 2024 2024-04-06T08:20:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:44 2024 2024-04-06T08:20:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ac6d51f9-12c0-4020-8ba6-c41cc112c761 2024-04-06T08:20:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_23 snap 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.export 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.export 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.export 2024-04-06T08:20:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.export 2024-04-06T08:20:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.export 2024-04-06T08:20:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.export 2024-04-06T08:20:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_23.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_23.export 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_24 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_24 snap 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:08.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-04-06T08:20:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-04-06T08:20:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-04-06T08:20:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T08:20:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-04-06T08:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-06T08:20:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-04-06T08:20:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-04-06T08:20:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_24.status 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_24.status 2024-04-06T08:20:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.status 2024-04-06T08:20:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_24.status 2024-04-06T08:20:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_24.status 2024-04-06T08:20:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-04-06T08:20:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-04-06T08:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 165 2024-04-06T08:20:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.status 2024-04-06T08:20:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.status 2024-04-06T08:20:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-04-06T08:20:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=165 2024-04-06T08:20:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-06T08:20:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=91 2024-04-06T08:20:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 165 = 91 2024-04-06T08:20:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-06T08:20:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 165 = '' 2024-04-06T08:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.status 2024-04-06T08:20:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-06T08:20:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=165 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 165 = 165 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-04-06T08:20:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_24': 2024-04-06T08:20:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1298d9401cfd 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1298d9401cfd 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:10:56 2024 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:10:56 2024 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:10:56 2024 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8f4562b6-b8d0-438b-a48f-defcceca733b 2024-04-06T08:20:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_24 snap 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.export 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.export 2024-04-06T08:20:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.export 2024-04-06T08:20:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.export 2024-04-06T08:20:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.export 2024-04-06T08:20:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.export 2024-04-06T08:20:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_24.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_24.export 2024-04-06T08:20:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_25 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_25 snap 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-04-06T08:20:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-04-06T08:20:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-04-06T08:20:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T08:20:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-04-06T08:20:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-04-06T08:20:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T08:20:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-04-06T08:20:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-04-06T08:20:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-06T08:20:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:20:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:20:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-04-06T08:20:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:18.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_25.status 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_25.status 2024-04-06T08:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.status 2024-04-06T08:20:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_25.status 2024-04-06T08:20:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_25.status 2024-04-06T08:20:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-04-06T08:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-04-06T08:20:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 167 2024-04-06T08:20:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.status 2024-04-06T08:20:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.status 2024-04-06T08:20:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-04-06T08:20:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=167 2024-04-06T08:20:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-06T08:20:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 167 = '' 2024-04-06T08:20:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-06T08:20:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 167 = '' 2024-04-06T08:20:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.status 2024-04-06T08:20:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-06T08:20:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=167 2024-04-06T08:20:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 167 = 167 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_25': 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de8d634aec 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de8d634aec 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:06 2024 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:06 2024 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:06 2024 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f2c3f78a-8f35-4a3c-b4ba-4152fd15fcbe 2024-04-06T08:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_25 snap 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.export 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.export 2024-04-06T08:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.export 2024-04-06T08:20:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.export 2024-04-06T08:20:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.export 2024-04-06T08:20:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.export 2024-04-06T08:20:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_25.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_25.export 2024-04-06T08:20:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_26 2024-04-06T08:20:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_26 snap 2024-04-06T08:20:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-04-06T08:20:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-04-06T08:20:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-04-06T08:20:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-06T08:20:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-04-06T08:20:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-06T08:20:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-04-06T08:20:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-04-06T08:20:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-06T08:20:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:20:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:20:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:20:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-04-06T08:20:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_26.status 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_26.status 2024-04-06T08:20:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.status 2024-04-06T08:20:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_26.status 2024-04-06T08:20:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_26.status 2024-04-06T08:20:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-04-06T08:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-04-06T08:20:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 169 2024-04-06T08:20:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.status 2024-04-06T08:20:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.status 2024-04-06T08:20:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-04-06T08:20:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=169 2024-04-06T08:20:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-06T08:20:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 169 = '' 2024-04-06T08:20:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-06T08:20:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 169 = '' 2024-04-06T08:20:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.status 2024-04-06T08:20:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-06T08:20:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=169 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 169 = 169 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_26': 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128611420da2 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128611420da2 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:19 2024 2024-04-06T08:20:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:19 2024 2024-04-06T08:20:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:19 2024 2024-04-06T08:20:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: e1986487-bd67-45b7-a50c-21436fe14d4d 2024-04-06T08:20:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_26 snap 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.export 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.export 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.export 2024-04-06T08:20:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.export 2024-04-06T08:20:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.export 2024-04-06T08:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.export 2024-04-06T08:20:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_26.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_26.export 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_27 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_27 snap 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:30.482 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:20:30.479+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d394d860 2024-04-06T08:20:30.482 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:20:30.479+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9d78c2750 2024-04-06T08:20:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-04-06T08:20:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-04-06T08:20:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-04-06T08:20:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-04-06T08:20:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T08:20:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-04-06T08:20:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-06T08:20:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-04-06T08:20:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_27.status 2024-04-06T08:20:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_27.status 2024-04-06T08:20:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.status 2024-04-06T08:20:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_27.status 2024-04-06T08:20:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_27.status 2024-04-06T08:20:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-04-06T08:20:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-04-06T08:20:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 171 2024-04-06T08:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.status 2024-04-06T08:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.status 2024-04-06T08:20:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-04-06T08:20:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=171 2024-04-06T08:20:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-06T08:20:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=94 2024-04-06T08:20:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 171 = 94 2024-04-06T08:20:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-06T08:20:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 171 = '' 2024-04-06T08:20:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.status 2024-04-06T08:20:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-06T08:20:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=171 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 171 = 171 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-04-06T08:20:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_27': 2024-04-06T08:20:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128fd648293e 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128fd648293e 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:29 2024 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:29 2024 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:29 2024 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8b927270-e88c-47d8-9f3e-375b09dbf2b2 2024-04-06T08:20:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_27 snap 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.export 2024-04-06T08:20:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.export 2024-04-06T08:20:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.export 2024-04-06T08:20:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.export 2024-04-06T08:20:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.export 2024-04-06T08:20:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.export 2024-04-06T08:20:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_27.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_27.export 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_28 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_28 snap 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-04-06T08:20:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-04-06T08:20:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-04-06T08:20:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T08:20:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-04-06T08:20:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T08:20:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-04-06T08:20:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-06T08:20:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2024-04-06T08:20:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:20:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:20:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:20:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:20:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2024-04-06T08:20:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_28.status 2024-04-06T08:20:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_28.status 2024-04-06T08:20:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.status 2024-04-06T08:20:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_28.status 2024-04-06T08:20:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_28.status 2024-04-06T08:20:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-04-06T08:20:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-04-06T08:20:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 173 2024-04-06T08:20:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.status 2024-04-06T08:20:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.status 2024-04-06T08:20:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-04-06T08:20:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=173 2024-04-06T08:20:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-06T08:20:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=95 2024-04-06T08:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 173 = 95 2024-04-06T08:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-06T08:20:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=95 2024-04-06T08:20:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 173 = 95 2024-04-06T08:20:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-06T08:20:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 173 = '' 2024-04-06T08:20:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:20:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.status 2024-04-06T08:20:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-06T08:20:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=173 2024-04-06T08:20:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 173 = 173 2024-04-06T08:20:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-04-06T08:20:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_28': 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12984f27ae6d 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12984f27ae6d 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:43 2024 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:43 2024 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:43 2024 2024-04-06T08:20:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 3c43d300-9de9-4d2a-ba10-0d577a7a18c0 2024-04-06T08:20:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_28 snap 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.export 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.export 2024-04-06T08:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.export 2024-04-06T08:20:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.export 2024-04-06T08:20:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.export 2024-04-06T08:20:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.export 2024-04-06T08:20:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_28.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_28.export 2024-04-06T08:20:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_29 2024-04-06T08:20:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_29 snap 2024-04-06T08:20:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-04-06T08:20:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-04-06T08:20:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-06T08:20:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-06T08:20:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T08:20:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-06T08:20:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T08:20:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-06T08:20:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-06T08:20:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-06T08:20:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:20:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:20:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:20:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:20:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:20:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-06T08:20:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_29.status 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_29.status 2024-04-06T08:20:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.status 2024-04-06T08:20:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_29.status 2024-04-06T08:20:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_29.status 2024-04-06T08:20:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-04-06T08:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-04-06T08:20:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 175 2024-04-06T08:20:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.status 2024-04-06T08:20:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.status 2024-04-06T08:20:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-04-06T08:20:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=175 2024-04-06T08:20:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-06T08:20:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 175 = '' 2024-04-06T08:20:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-06T08:20:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 175 = '' 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.status 2024-04-06T08:20:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-06T08:20:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=175 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 175 = 175 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_29': 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10def23251bc 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10def23251bc 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:20:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:11:55 2024 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:11:55 2024 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:11:55 2024 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: c5b8cfe3-a440-4aa6-b7ed-bbf8e31b2175 2024-04-06T08:20:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_29 snap 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.export 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.export 2024-04-06T08:20:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.export 2024-04-06T08:20:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.export 2024-04-06T08:20:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.export 2024-04-06T08:20:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:20:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.export 2024-04-06T08:20:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_29.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_29.export 2024-04-06T08:20:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:20:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_30 2024-04-06T08:20:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_30 snap 2024-04-06T08:20:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:20:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:20:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:20:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:20:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-04-06T08:20:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:20:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:20:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:20:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:20:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:20:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:20:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-04-06T08:20:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:20:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:20:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:20:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:20:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:20:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:20:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:20:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:20:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-04-06T08:20:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T08:20:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:20:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:20:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:20:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:20:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:20:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:20:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:20:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-04-06T08:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T08:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-04-06T08:20:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:20:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:20:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:20:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:20:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:20:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-04-06T08:20:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-06T08:20:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:20:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:20:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:20:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2024-04-06T08:20:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:20:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:20:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:20:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:20:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:20:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2024-04-06T08:20:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:20:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:20:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:20:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:20:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_30.status 2024-04-06T08:20:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_30.status 2024-04-06T08:20:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.status 2024-04-06T08:20:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_30.status 2024-04-06T08:20:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_30.status 2024-04-06T08:20:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:20:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-04-06T08:20:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-04-06T08:20:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 177 2024-04-06T08:20:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.status 2024-04-06T08:20:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:20:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.status 2024-04-06T08:20:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-04-06T08:20:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:20:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=177 2024-04-06T08:20:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:20:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-06T08:20:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=97 2024-04-06T08:20:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 177 = 97 2024-04-06T08:20:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:20:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-06T08:20:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 177 = '' 2024-04-06T08:20:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:20:58.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5677550075785572) > weight (0.25) 2024-04-06T08:20:58.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4249514592244025) > weight (0.25) 2024-04-06T08:20:58.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.38718471228733053) > weight (0.25) 2024-04-06T08:20:58.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.465532721850314) > weight (0.25) 2024-04-06T08:20:58.187 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-04-06T08:20:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:20:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:20:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:20:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:20:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-06T08:20:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:20:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:20:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:20:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:20:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:20:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:20:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:20:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 177 = '' 2024-04-06T08:20:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:20:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:21:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:21:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:21:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.status 2024-04-06T08:21:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-06T08:21:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=177 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 177 = 177 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:21:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-04-06T08:21:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_30': 2024-04-06T08:21:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:21:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:21:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:21:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128666de6025 2024-04-06T08:21:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128666de6025 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:01 2024 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:01 2024 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:01 2024 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 0ac3adc2-3f86-4a07-a54a-4fd87c334a97 2024-04-06T08:21:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_30 snap 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.export 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.export 2024-04-06T08:21:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.export 2024-04-06T08:21:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.export 2024-04-06T08:21:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.export 2024-04-06T08:21:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.export 2024-04-06T08:21:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_30.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_30.export 2024-04-06T08:21:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:21:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_31 2024-04-06T08:21:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_31 snap 2024-04-06T08:21:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:21:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-04-06T08:21:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:21:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:21:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:21:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-04-06T08:21:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-04-06T08:21:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-06T08:21:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-04-06T08:21:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-04-06T08:21:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:21:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:21:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:21:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:21:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:21:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:21:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:21:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_31.status 2024-04-06T08:21:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_31.status 2024-04-06T08:21:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:21:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.status 2024-04-06T08:21:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_31.status 2024-04-06T08:21:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:21:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_31.status 2024-04-06T08:21:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-04-06T08:21:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-04-06T08:21:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 179 2024-04-06T08:21:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.status 2024-04-06T08:21:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.status 2024-04-06T08:21:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-04-06T08:21:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:21:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=179 2024-04-06T08:21:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:21:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-06T08:21:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=98 2024-04-06T08:21:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 179 = 98 2024-04-06T08:21:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:21:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-06T08:21:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:21:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:21:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:21:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:21:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 179 = '' 2024-04-06T08:21:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:21:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.status 2024-04-06T08:21:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-06T08:21:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=179 2024-04-06T08:21:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 179 = 179 2024-04-06T08:21:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-04-06T08:21:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:21:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_31': 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128fd393170 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128fd393170 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:21:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:16 2024 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:16 2024 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:16 2024 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: a12859fc-49ad-4646-9f39-4212f7cb141b 2024-04-06T08:21:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:21:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_31 snap 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.export 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.export 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.export 2024-04-06T08:21:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.export 2024-04-06T08:21:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.export 2024-04-06T08:21:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.export 2024-04-06T08:21:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_31.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_31.export 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_32 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_32 snap 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:21:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:21:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:21:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-04-06T08:21:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:21:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:21:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-04-06T08:21:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-06T08:21:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-04-06T08:21:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:21:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:21:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:21:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:21:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:21:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-04-06T08:21:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:21:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_32.status 2024-04-06T08:21:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_32.status 2024-04-06T08:21:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:21:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.status 2024-04-06T08:21:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_32.status 2024-04-06T08:21:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:21:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_32.status 2024-04-06T08:21:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-04-06T08:21:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-04-06T08:21:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 181 2024-04-06T08:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.status 2024-04-06T08:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.status 2024-04-06T08:21:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-04-06T08:21:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:21:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=181 2024-04-06T08:21:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:21:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-06T08:21:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=99 2024-04-06T08:21:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 181 = 99 2024-04-06T08:21:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:21:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-06T08:21:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:21:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:21:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:21:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:21:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 181 = '' 2024-04-06T08:21:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:21:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.status 2024-04-06T08:21:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-06T08:21:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=181 2024-04-06T08:21:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 181 = 181 2024-04-06T08:21:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:21:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-04-06T08:21:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_32': 2024-04-06T08:21:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:21:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12a8b6267d4e 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12a8b6267d4e 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:31 2024 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:31 2024 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:31 2024 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 079afbf7-3e02-4a0c-b385-2d20983af07c 2024-04-06T08:21:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:21:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_32 snap 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.export 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.export 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.export 2024-04-06T08:21:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.export 2024-04-06T08:21:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.export 2024-04-06T08:21:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.export 2024-04-06T08:21:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_32.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_32.export 2024-04-06T08:21:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:21:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_33 2024-04-06T08:21:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_33 snap 2024-04-06T08:21:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:21:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:19.087 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.25243472933301925) > weight (0.25) 2024-04-06T08:21:19.087 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-06T08:21:19.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-06T08:21:19.088 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T08:21:19.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:21:19.090+0000 7f26b2456640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48141) UID: 1000 2024-04-06T08:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-04-06T08:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:21:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:21:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-06T08:21:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:21:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:21:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:21:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:21:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-06T08:21:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:21:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:21:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:21:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-06T08:21:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 1 2024-04-06T08:21:21.089 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-06T08:21:21.089 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:21:21.090 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-06T08:21:21.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T08:21:21.090 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T08:21:21.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T08:21:21.093 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-04-06T08:21:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:21:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:21:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-06T08:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-06T08:21:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:21:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:21:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:21:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-06T08:21:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 2 2024-04-06T08:21:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:21:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:21:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:21:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-06T08:21:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:21:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:21:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:21:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-06T08:21:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 4 2024-04-06T08:21:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-06T08:21:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:21:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:21:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:21:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-06T08:21:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:21:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:21:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:21:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:21:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:21:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-06T08:21:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 8 2024-04-06T08:21:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-06T08:21:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:21:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:21:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-06T08:21:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:21:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:21:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:21:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:21:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:21:37.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:21:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-06T08:21:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 8 2024-04-06T08:21:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:21:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:21:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:21:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:21:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-06T08:21:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-06T08:21:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:21:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:21:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:21:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:21:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:21:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-06T08:21:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:21:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 8 2024-04-06T08:21:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:21:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:21:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:21:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:21:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:21:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:21:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-06T08:21:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-06T08:21:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:21:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:21:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:21:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-06T08:21:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:21:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:21:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:21:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:21:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:21:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:21:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:21:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:21:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:21:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:21:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:21:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:21:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:21:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:21:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:21:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:21:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:21:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:21:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:21:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:21:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:21:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:21:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:21:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-06T08:21:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:21:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:21:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:21:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:21:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:21:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:21:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:21:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_33.status 2024-04-06T08:21:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_33.status 2024-04-06T08:21:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:21:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.status 2024-04-06T08:21:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_33.status 2024-04-06T08:21:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_33.status 2024-04-06T08:21:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:21:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-04-06T08:21:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-04-06T08:21:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 183 2024-04-06T08:21:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.status 2024-04-06T08:21:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.status 2024-04-06T08:21:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-04-06T08:21:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:21:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=183 2024-04-06T08:21:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:21:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-04-06T08:21:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:21:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:21:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:21:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:21:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 183 = '' 2024-04-06T08:21:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:21:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-04-06T08:21:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:21:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:21:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:21:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:21:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 183 = '' 2024-04-06T08:21:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:21:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.status 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-04-06T08:21:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:21:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=183 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 183 = 183 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:21:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:21:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:21:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-04-06T08:21:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_33': 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12a820bca412 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12a820bca412 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:45 2024 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:45 2024 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:45 2024 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:21:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:21:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 43b614ea-0108-4b77-ae7f-5b1e2003fce5 2024-04-06T08:21:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:21:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_33 snap 2024-04-06T08:21:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:21:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:21:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.export 2024-04-06T08:21:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.export 2024-04-06T08:21:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.export 2024-04-06T08:21:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.export 2024-04-06T08:21:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.export 2024-04-06T08:21:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:21:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.export 2024-04-06T08:21:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_33.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_33.export 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_34 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_34 snap 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:21:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-04-06T08:22:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-04-06T08:22:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:01.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-04-06T08:22:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T08:22:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:22:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:22:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:22:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:22:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:22:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-04-06T08:22:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-06T08:22:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-04-06T08:22:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:22:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:22:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:22:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:22:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-04-06T08:22:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-04-06T08:22:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_34.status 2024-04-06T08:22:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_34.status 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.status 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_34.status 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_34.status 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-04-06T08:22:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 185 2024-04-06T08:22:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.status 2024-04-06T08:22:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:02.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.status 2024-04-06T08:22:02.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:02.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-04-06T08:22:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=185 2024-04-06T08:22:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-04-06T08:22:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=101 2024-04-06T08:22:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 185 = 101 2024-04-06T08:22:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-04-06T08:22:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 185 = '' 2024-04-06T08:22:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.status 2024-04-06T08:22:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-04-06T08:22:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=185 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 185 = 185 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-04-06T08:22:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_34': 2024-04-06T08:22:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1286cd4e842f 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1286cd4e842f 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:12:59 2024 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:12:59 2024 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:12:59 2024 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 6d663376-e7fa-4e9d-b581-641ed5f5b057 2024-04-06T08:22:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_34 snap 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.export 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.export 2024-04-06T08:22:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.export 2024-04-06T08:22:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.export 2024-04-06T08:22:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.export 2024-04-06T08:22:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.export 2024-04-06T08:22:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_34.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_34.export 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_35 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_35 snap 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-04-06T08:22:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-04-06T08:22:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-04-06T08:22:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-04-06T08:22:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-04-06T08:22:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_35.status 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_35.status 2024-04-06T08:22:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.status 2024-04-06T08:22:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_35.status 2024-04-06T08:22:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_35.status 2024-04-06T08:22:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-04-06T08:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-04-06T08:22:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 187 2024-04-06T08:22:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.status 2024-04-06T08:22:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.status 2024-04-06T08:22:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-04-06T08:22:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=187 2024-04-06T08:22:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-06T08:22:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 187 = '' 2024-04-06T08:22:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-06T08:22:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 187 = '' 2024-04-06T08:22:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.status 2024-04-06T08:22:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-06T08:22:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=187 2024-04-06T08:22:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 187 = 187 2024-04-06T08:22:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_35': 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128f1dd4b939 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128f1dd4b939 2024-04-06T08:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:13 2024 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:13 2024 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:13 2024 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8219f002-4b2d-40e9-9cd8-ae83bc156dc4 2024-04-06T08:22:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_35 snap 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.export 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.export 2024-04-06T08:22:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.export 2024-04-06T08:22:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.export 2024-04-06T08:22:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.export 2024-04-06T08:22:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.export 2024-04-06T08:22:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_35.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_35.export 2024-04-06T08:22:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_36 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_36 snap 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-04-06T08:22:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T08:22:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:22:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:22:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-04-06T08:22:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T08:22:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-04-06T08:22:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:22:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:22:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:22:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:22:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:22:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-04-06T08:22:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-06T08:22:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-04-06T08:22:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:22:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:22:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:22:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:22:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:22:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:22:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:22:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:22:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:22:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-04-06T08:22:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_36.status 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_36.status 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.status 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_36.status 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_36.status 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-04-06T08:22:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-04-06T08:22:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 189 2024-04-06T08:22:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.status 2024-04-06T08:22:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.status 2024-04-06T08:22:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-04-06T08:22:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=189 2024-04-06T08:22:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-06T08:22:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=103 2024-04-06T08:22:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 189 = 103 2024-04-06T08:22:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-06T08:22:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 189 = '' 2024-04-06T08:22:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-06T08:22:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 189 = '' 2024-04-06T08:22:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:22:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.status 2024-04-06T08:22:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-06T08:22:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=189 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 189 = 189 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-04-06T08:22:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_36': 2024-04-06T08:22:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12a834489f34 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12a834489f34 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:26 2024 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:26 2024 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:26 2024 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 111e48f6-921e-4a32-9d71-4f87dd955b5e 2024-04-06T08:22:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_36 snap 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.export 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.export 2024-04-06T08:22:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.export 2024-04-06T08:22:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.export 2024-04-06T08:22:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.export 2024-04-06T08:22:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.export 2024-04-06T08:22:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_36.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_36.export 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_37 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_37 snap 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-04-06T08:22:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-04-06T08:22:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-04-06T08:22:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T08:22:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:22:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:22:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:22:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-04-06T08:22:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-06T08:22:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:22:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:25.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:22:25.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:22:25.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:22:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-04-06T08:22:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_37.status 2024-04-06T08:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_37.status 2024-04-06T08:22:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.status 2024-04-06T08:22:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_37.status 2024-04-06T08:22:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_37.status 2024-04-06T08:22:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-04-06T08:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-04-06T08:22:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 191 2024-04-06T08:22:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.status 2024-04-06T08:22:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.status 2024-04-06T08:22:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-04-06T08:22:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=191 2024-04-06T08:22:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:26.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-04-06T08:22:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=104 2024-04-06T08:22:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 191 = 104 2024-04-06T08:22:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-04-06T08:22:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:27.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 191 = '' 2024-04-06T08:22:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.status 2024-04-06T08:22:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-04-06T08:22:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=191 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 191 = 191 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_37': 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de37198767 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de37198767 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:39 2024 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:39 2024 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:39 2024 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 9c0cee78-3594-4b38-aea6-9da642e815a7 2024-04-06T08:22:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_37 snap 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.export 2024-04-06T08:22:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.export 2024-04-06T08:22:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.export 2024-04-06T08:22:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.export 2024-04-06T08:22:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.export 2024-04-06T08:22:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.export 2024-04-06T08:22:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_37.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_37.export 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_38 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_38 snap 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-04-06T08:22:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-04-06T08:22:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-04-06T08:22:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_38.status 2024-04-06T08:22:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_38.status 2024-04-06T08:22:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.status 2024-04-06T08:22:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_38.status 2024-04-06T08:22:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_38.status 2024-04-06T08:22:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-04-06T08:22:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-04-06T08:22:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 193 2024-04-06T08:22:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.status 2024-04-06T08:22:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.status 2024-04-06T08:22:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-04-06T08:22:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=193 2024-04-06T08:22:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-04-06T08:22:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=105 2024-04-06T08:22:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 193 = 105 2024-04-06T08:22:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-04-06T08:22:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 193 = '' 2024-04-06T08:22:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.status 2024-04-06T08:22:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-04-06T08:22:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=193 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 193 = 193 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-04-06T08:22:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_38': 2024-04-06T08:22:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12867ef2fd78 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12867ef2fd78 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:13:49 2024 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:13:49 2024 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:13:49 2024 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: bee07777-cb98-40aa-a6a6-2b3e5d90cb96 2024-04-06T08:22:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_38 snap 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.export 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.export 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.export 2024-04-06T08:22:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.export 2024-04-06T08:22:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.export 2024-04-06T08:22:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.export 2024-04-06T08:22:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_38.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_38.export 2024-04-06T08:22:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_39 2024-04-06T08:22:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_39 snap 2024-04-06T08:22:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:38.188 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T08:22:38.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T08:22:38.189 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-06T08:22:38.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:22:38.188+0000 7ff955b70640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 47121) UID: 1000 2024-04-06T08:22:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-04-06T08:22:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-04-06T08:22:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-04-06T08:22:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:40.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-04-06T08:22:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T08:22:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-04-06T08:22:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:22:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:22:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:22:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:22:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-04-06T08:22:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T08:22:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-04-06T08:22:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:22:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:22:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:22:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:22:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-04-06T08:22:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-06T08:22:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2024-04-06T08:22:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:22:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:22:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:22:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:22:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:22:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:22:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:22:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2024-04-06T08:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_39.status 2024-04-06T08:22:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_39.status 2024-04-06T08:22:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.status 2024-04-06T08:22:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_39.status 2024-04-06T08:22:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_39.status 2024-04-06T08:22:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-04-06T08:22:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-04-06T08:22:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 195 2024-04-06T08:22:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.status 2024-04-06T08:22:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.status 2024-04-06T08:22:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-04-06T08:22:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=195 2024-04-06T08:22:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-06T08:22:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=106 2024-04-06T08:22:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 195 = 106 2024-04-06T08:22:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-06T08:22:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=106 2024-04-06T08:22:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 195 = 106 2024-04-06T08:22:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-06T08:22:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 195 = '' 2024-04-06T08:22:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:22:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.status 2024-04-06T08:22:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-06T08:22:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=195 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 195 = 195 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:45.189 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T08:22:45.190 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:22:45.190 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T08:22:45.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T08:22:45.190 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T08:22:45.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T08:22:45.193 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-06T08:22:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_39': 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128fdf22f1d 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128fdf22f1d 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:03 2024 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:03 2024 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:03 2024 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 2f88af95-4f57-486d-93b3-f203c018dd1a 2024-04-06T08:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_39 snap 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.export 2024-04-06T08:22:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.export 2024-04-06T08:22:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.export 2024-04-06T08:22:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.export 2024-04-06T08:22:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.export 2024-04-06T08:22:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.export 2024-04-06T08:22:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_39.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_39.export 2024-04-06T08:22:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_40 2024-04-06T08:22:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_40 snap 2024-04-06T08:22:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-04-06T08:22:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:48.478 INFO:tasks.ceph.cluster1.osd.2.smithi047.stderr:2024-04-06T08:22:48.475+0000 7f48e4ffb640 -1 reset not still connected to 0x55a9da2fedd0 2024-04-06T08:22:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-04-06T08:22:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-04-06T08:22:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-04-06T08:22:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-06T08:22:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-04-06T08:22:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:22:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:22:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:22:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-04-06T08:22:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-04-06T08:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_40.status 2024-04-06T08:22:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_40.status 2024-04-06T08:22:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.status 2024-04-06T08:22:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_40.status 2024-04-06T08:22:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_40.status 2024-04-06T08:22:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-04-06T08:22:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-04-06T08:22:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 197 2024-04-06T08:22:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.status 2024-04-06T08:22:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.status 2024-04-06T08:22:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-04-06T08:22:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=197 2024-04-06T08:22:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-04-06T08:22:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 197 = '' 2024-04-06T08:22:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-04-06T08:22:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 197 = '' 2024-04-06T08:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.status 2024-04-06T08:22:51.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-04-06T08:22:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=197 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 197 = 197 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:22:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_40': 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12a858778fff 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12a858778fff 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:22:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:10 2024 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:10 2024 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:10 2024 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 3436f7ad-064f-4e05-bbf9-791ed79dfc9d 2024-04-06T08:22:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_40 snap 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.export 2024-04-06T08:22:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.export 2024-04-06T08:22:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.export 2024-04-06T08:22:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.export 2024-04-06T08:22:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.export 2024-04-06T08:22:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:22:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.export 2024-04-06T08:22:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_40.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_40.export 2024-04-06T08:22:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:22:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_41 2024-04-06T08:22:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_41 snap 2024-04-06T08:22:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:22:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:22:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-04-06T08:22:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:22:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:22:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:22:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:22:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:22:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:22:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:22:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:22:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:22:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-04-06T08:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-06T08:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:22:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:22:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-04-06T08:22:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-06T08:22:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:22:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:22:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:22:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:22:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:22:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:22:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-04-06T08:22:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:22:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:22:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:22:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:22:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:22:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:22:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_41.status 2024-04-06T08:22:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_41.status 2024-04-06T08:22:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.status 2024-04-06T08:22:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_41.status 2024-04-06T08:22:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_41.status 2024-04-06T08:22:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-04-06T08:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-04-06T08:22:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 199 2024-04-06T08:22:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.status 2024-04-06T08:22:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:22:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.status 2024-04-06T08:22:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-04-06T08:22:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:22:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=199 2024-04-06T08:22:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:22:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-06T08:22:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=108 2024-04-06T08:22:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 199 = 108 2024-04-06T08:22:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:22:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-06T08:22:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 199 = '' 2024-04-06T08:22:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:22:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:22:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:22:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:22:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:22:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-06T08:22:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:22:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:22:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:22:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:22:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:22:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:22:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:22:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 199 = '' 2024-04-06T08:22:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:22:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:23:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:23:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:23:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.status 2024-04-06T08:23:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-06T08:23:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=199 2024-04-06T08:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 199 = 199 2024-04-06T08:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-04-06T08:23:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-04-06T08:23:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_41': 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de365360dd 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de365360dd 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:17 2024 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:17 2024 2024-04-06T08:23:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:17 2024 2024-04-06T08:23:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 175e0311-52c5-4f3f-9e73-a36267e32a86 2024-04-06T08:23:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_41 snap 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.export 2024-04-06T08:23:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.export 2024-04-06T08:23:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.export 2024-04-06T08:23:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.export 2024-04-06T08:23:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.export 2024-04-06T08:23:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.export 2024-04-06T08:23:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_41.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_41.export 2024-04-06T08:23:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_42 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_42 snap 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:03.093 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4689575959595359) > weight (0.25) 2024-04-06T08:23:03.094 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-06T08:23:03.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-06T08:23:03.094 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T08:23:03.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:23:03.092+0000 7f2de290a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 50527) UID: 1000 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-04-06T08:23:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:05.095 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-06T08:23:05.095 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:23:05.095 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-06T08:23:05.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T08:23:05.096 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T08:23:05.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T08:23:05.098 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-04-06T08:23:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-04-06T08:23:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_42.status 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_42.status 2024-04-06T08:23:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.status 2024-04-06T08:23:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_42.status 2024-04-06T08:23:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_42.status 2024-04-06T08:23:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-04-06T08:23:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-04-06T08:23:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 201 2024-04-06T08:23:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.status 2024-04-06T08:23:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.status 2024-04-06T08:23:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-04-06T08:23:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=201 2024-04-06T08:23:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-06T08:23:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=109 2024-04-06T08:23:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 201 = 109 2024-04-06T08:23:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-06T08:23:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=109 2024-04-06T08:23:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 201 = 109 2024-04-06T08:23:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-06T08:23:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 201 = '' 2024-04-06T08:23:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:23:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.status 2024-04-06T08:23:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-06T08:23:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=201 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 201 = 201 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_42': 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12b886b15e09 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12b886b15e09 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:24 2024 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:24 2024 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:24 2024 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f200c959-0115-400f-90ae-1785585fdbef 2024-04-06T08:23:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_42 snap 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.export 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.export 2024-04-06T08:23:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.export 2024-04-06T08:23:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.export 2024-04-06T08:23:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.export 2024-04-06T08:23:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.export 2024-04-06T08:23:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_42.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_42.export 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_43 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_43 snap 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-04-06T08:23:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-04-06T08:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-06T08:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-04-06T08:23:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:23:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:23:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-04-06T08:23:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-06T08:23:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:23:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:23:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:23:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:23:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-04-06T08:23:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_43.status 2024-04-06T08:23:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_43.status 2024-04-06T08:23:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.status 2024-04-06T08:23:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_43.status 2024-04-06T08:23:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_43.status 2024-04-06T08:23:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-04-06T08:23:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-04-06T08:23:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 203 2024-04-06T08:23:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.status 2024-04-06T08:23:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.status 2024-04-06T08:23:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-04-06T08:23:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=203 2024-04-06T08:23:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-06T08:23:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 203 = '' 2024-04-06T08:23:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-06T08:23:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 203 = '' 2024-04-06T08:23:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.status 2024-04-06T08:23:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-06T08:23:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=203 2024-04-06T08:23:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 203 = 203 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_43': 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de41a3a5e4 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de41a3a5e4 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:35 2024 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:35 2024 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:35 2024 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 1a6528d8-9d7a-4762-8686-78b7552a3f94 2024-04-06T08:23:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_43 snap 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.export 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.export 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.export 2024-04-06T08:23:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.export 2024-04-06T08:23:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.export 2024-04-06T08:23:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.export 2024-04-06T08:23:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_43.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_43.export 2024-04-06T08:23:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_44 2024-04-06T08:23:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_44 snap 2024-04-06T08:23:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-04-06T08:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-04-06T08:23:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-04-06T08:23:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-06T08:23:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:23:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:23:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:23:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:23:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-04-06T08:23:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_44.status 2024-04-06T08:23:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_44.status 2024-04-06T08:23:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.status 2024-04-06T08:23:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_44.status 2024-04-06T08:23:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_44.status 2024-04-06T08:23:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-04-06T08:23:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-04-06T08:23:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 205 2024-04-06T08:23:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.status 2024-04-06T08:23:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.status 2024-04-06T08:23:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-04-06T08:23:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=205 2024-04-06T08:23:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-06T08:23:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=111 2024-04-06T08:23:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 205 = 111 2024-04-06T08:23:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:23.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-06T08:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=111 2024-04-06T08:23:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 205 = 111 2024-04-06T08:23:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-06T08:23:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 205 = '' 2024-04-06T08:23:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:23:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.status 2024-04-06T08:23:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-06T08:23:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=205 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 205 = 205 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-04-06T08:23:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_44': 2024-04-06T08:23:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12b87b324890 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12b87b324890 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:14:48 2024 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:14:48 2024 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:14:48 2024 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 79a01d9a-fb10-4dfa-97e4-0874260e4dde 2024-04-06T08:23:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_44 snap 2024-04-06T08:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.export 2024-04-06T08:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.export 2024-04-06T08:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.export 2024-04-06T08:23:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.export 2024-04-06T08:23:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:27.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.export 2024-04-06T08:23:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.export 2024-04-06T08:23:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_44.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_44.export 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_45 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_45 snap 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-04-06T08:23:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-04-06T08:23:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T08:23:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:23:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:23:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:23:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-04-06T08:23:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-06T08:23:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:23:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:23:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:23:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-04-06T08:23:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_45.status 2024-04-06T08:23:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_45.status 2024-04-06T08:23:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.status 2024-04-06T08:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_45.status 2024-04-06T08:23:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_45.status 2024-04-06T08:23:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-04-06T08:23:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-04-06T08:23:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 207 2024-04-06T08:23:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.status 2024-04-06T08:23:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.status 2024-04-06T08:23:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-04-06T08:23:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:32.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=207 2024-04-06T08:23:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-04-06T08:23:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=112 2024-04-06T08:23:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 207 = 112 2024-04-06T08:23:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-04-06T08:23:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 207 = '' 2024-04-06T08:23:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.status 2024-04-06T08:23:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-04-06T08:23:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=207 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 207 = 207 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-04-06T08:23:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_45': 2024-04-06T08:23:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10dea2cf5d07 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10dea2cf5d07 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:00 2024 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:00 2024 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:00 2024 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 5ba3668d-93d2-4d22-87c3-f63064c87646 2024-04-06T08:23:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_45 snap 2024-04-06T08:23:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.export 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.export 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.export 2024-04-06T08:23:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.export 2024-04-06T08:23:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.export 2024-04-06T08:23:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.export 2024-04-06T08:23:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_45.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_45.export 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_46 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_46 snap 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:35.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-04-06T08:23:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-04-06T08:23:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-04-06T08:23:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T08:23:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:23:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:23:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-04-06T08:23:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T08:23:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-04-06T08:23:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:23:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:23:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:23:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:23:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:23:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-04-06T08:23:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-06T08:23:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:23:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:23:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:23:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:23:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:23:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:23:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:23:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:23:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:23:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-04-06T08:23:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_46.status 2024-04-06T08:23:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_46.status 2024-04-06T08:23:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.status 2024-04-06T08:23:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_46.status 2024-04-06T08:23:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_46.status 2024-04-06T08:23:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-04-06T08:23:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-04-06T08:23:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 209 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.status 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.status 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-04-06T08:23:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=209 2024-04-06T08:23:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-06T08:23:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 209 = '' 2024-04-06T08:23:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-06T08:23:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 209 = '' 2024-04-06T08:23:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.status 2024-04-06T08:23:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-06T08:23:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=209 2024-04-06T08:23:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 209 = 209 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-04-06T08:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_46': 2024-04-06T08:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1286e0622a57 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1286e0622a57 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:07 2024 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:07 2024 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:07 2024 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 5084cde1-4c2c-469a-b871-05d78b2a8610 2024-04-06T08:23:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_46 snap 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.export 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.export 2024-04-06T08:23:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.export 2024-04-06T08:23:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.export 2024-04-06T08:23:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.export 2024-04-06T08:23:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.export 2024-04-06T08:23:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_46.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_46.export 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_47 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_47 snap 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-04-06T08:23:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:46.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:46.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-04-06T08:23:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-06T08:23:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:23:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:23:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:23:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:23:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:23:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:23:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:23:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-04-06T08:23:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-06T08:23:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:23:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:23:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:23:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:23:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-04-06T08:23:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-06T08:23:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:23:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:23:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:23:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:23:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:23:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:23:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-04-06T08:23:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_47.status 2024-04-06T08:23:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_47.status 2024-04-06T08:23:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.status 2024-04-06T08:23:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_47.status 2024-04-06T08:23:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_47.status 2024-04-06T08:23:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-04-06T08:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-04-06T08:23:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 211 2024-04-06T08:23:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.status 2024-04-06T08:23:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.status 2024-04-06T08:23:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-04-06T08:23:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=211 2024-04-06T08:23:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:47.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.40085386991854755) > weight (0.25) 2024-04-06T08:23:47.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7728896968213834) > weight (0.25) 2024-04-06T08:23:47.194 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-06T08:23:47.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-06T08:23:47.195 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-06T08:23:47.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:23:47.195+0000 7f7d10955640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42199) UID: 1000 2024-04-06T08:23:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-06T08:23:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=114 2024-04-06T08:23:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 211 = 114 2024-04-06T08:23:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-06T08:23:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 211 = '' 2024-04-06T08:23:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-06T08:23:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 211 = '' 2024-04-06T08:23:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:23:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.status 2024-04-06T08:23:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-06T08:23:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=211 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 211 = 211 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-04-06T08:23:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_47': 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 128fe51f8651 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.128fe51f8651 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:15 2024 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:15 2024 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:15 2024 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:23:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:23:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 339c1583-8140-4531-ae3a-4bb2efc43b9e 2024-04-06T08:23:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_47 snap 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.export 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.export 2024-04-06T08:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.export 2024-04-06T08:23:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.export 2024-04-06T08:23:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.export 2024-04-06T08:23:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:23:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.export 2024-04-06T08:23:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_47.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_47.export 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_48 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_48 snap 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:23:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-04-06T08:23:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:23:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:23:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-04-06T08:23:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:23:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:23:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:23:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:23:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:23:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:23:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:23:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:23:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:23:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:23:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-04-06T08:23:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:23:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:23:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:23:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:23:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:23:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:23:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-04-06T08:23:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:23:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:23:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:23:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:23:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:23:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:23:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_48.status 2024-04-06T08:23:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_48.status 2024-04-06T08:23:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.status 2024-04-06T08:23:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_48.status 2024-04-06T08:23:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_48.status 2024-04-06T08:23:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:23:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-04-06T08:23:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-04-06T08:23:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 213 2024-04-06T08:23:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.status 2024-04-06T08:23:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:23:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.status 2024-04-06T08:23:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-04-06T08:23:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:23:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=213 2024-04-06T08:23:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:23:56.196 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-06T08:23:56.196 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:23:56.196 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-06T08:23:56.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-06T08:23:56.197 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-06T08:23:56.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-06T08:23:56.199 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-06T08:23:56.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=115 2024-04-06T08:23:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 213 = 115 2024-04-06T08:23:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:23:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-06T08:23:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=115 2024-04-06T08:23:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 213 = 115 2024-04-06T08:23:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-06T08:23:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:23:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:23:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:23:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:23:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 213 = '' 2024-04-06T08:23:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:23:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:23:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.status 2024-04-06T08:23:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-06T08:23:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:23:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=213 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 213 = 213 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:24:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-04-06T08:24:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_48': 2024-04-06T08:24:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:24:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:24:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12b816fae223 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12b816fae223 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:22 2024 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:22 2024 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:22 2024 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: ed498893-396c-4438-80fa-19e53e8f4471 2024-04-06T08:24:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_48 snap 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.export 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.export 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.export 2024-04-06T08:24:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.export 2024-04-06T08:24:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:24:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.export 2024-04-06T08:24:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:24:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.export 2024-04-06T08:24:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_48.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_48.export 2024-04-06T08:24:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_49 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_49 snap 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:24:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:24:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:24:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:24:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-04-06T08:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:24:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:24:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:24:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:24:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:24:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:24:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-04-06T08:24:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:24:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:24:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:24:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:24:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:24:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:24:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_49.status 2024-04-06T08:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_49.status 2024-04-06T08:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.status 2024-04-06T08:24:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_49.status 2024-04-06T08:24:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_49.status 2024-04-06T08:24:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:24:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-04-06T08:24:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-04-06T08:24:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 215 2024-04-06T08:24:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.status 2024-04-06T08:24:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.status 2024-04-06T08:24:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-04-06T08:24:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:24:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=215 2024-04-06T08:24:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:24:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:24:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-06T08:24:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=116 2024-04-06T08:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 215 = 116 2024-04-06T08:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:24:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-06T08:24:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=116 2024-04-06T08:24:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 215 = 116 2024-04-06T08:24:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-06T08:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:24:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:24:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:24:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:24:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:24:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 215 = '' 2024-04-06T08:24:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:24:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.status 2024-04-06T08:24:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-06T08:24:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=215 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 215 = 215 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:24:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:24:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-04-06T08:24:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_49': 2024-04-06T08:24:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 10de5bf70331 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.10de5bf70331 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:30 2024 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:30 2024 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:30 2024 2024-04-06T08:24:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:24:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:24:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:24:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: f62960e8-a1e8-461c-bf37-b3f594ac8f8b 2024-04-06T08:24:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:24:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_49 snap 2024-04-06T08:24:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:24:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:24:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.export 2024-04-06T08:24:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.export 2024-04-06T08:24:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.export 2024-04-06T08:24:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.export 2024-04-06T08:24:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:24:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.export 2024-04-06T08:24:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:24:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.export 2024-04-06T08:24:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_49.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_49.export 2024-04-06T08:24:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_50 2024-04-06T08:24:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_snap cluster2 mirror image_50 snap 2024-04-06T08:24:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=started 2024-04-06T08:24:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:24:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:24:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=started 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_result 2024-04-06T08:24:13.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=stopped 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_instance=cluster1 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -i state 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance=cluster1 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local loop_instance 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-06T08:24:13.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sleep 0 2024-04-06T08:24:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-06T08:24:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ seq 0 3 2024-04-06T08:24:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:0 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:0 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=0 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:0 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:0 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=0 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:24:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:24:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=0 2024-04-06T08:24:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=0 2024-04-06T08:24:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-06T08:24:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:24:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:24:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-06T08:24:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-06T08:24:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-06T08:24:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:24:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:1 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:1 2024-04-06T08:24:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-06T08:24:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=1 2024-04-06T08:24:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-06T08:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-06T08:24:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-06T08:24:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-06T08:24:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:24:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:24:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:2 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:2 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=2 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=2 2024-04-06T08:24:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:2 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:2 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=2 2024-04-06T08:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-06T08:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=2 2024-04-06T08:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=2 2024-04-06T08:24:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-06T08:24:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:24:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:24:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-06T08:24:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-04-06T08:24:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-06T08:24:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd mirror status : get status for rbd mirror 2024-04-06T08:24:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:admin_socket: invalid command 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1:3 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ shift 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local val=cluster1:3 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster_var_name=cluster 2024-04-06T08:24:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local instance_var_name=instance 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=3 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval cluster=cluster1 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ eval instance=3 2024-04-06T08:24:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:24:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-06T08:24:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local local_cluster=cluster1:3 2024-04-06T08:24:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster=cluster1 2024-04-06T08:24:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local val=cluster1:3 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local cluster_var_name=local_cluster 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ local instance_var_name=instance 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ cluster=cluster1 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ instance=3 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval local_cluster=cluster1 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ local_cluster=cluster1 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ eval instance=3 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ instance=3 2024-04-06T08:24:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-06T08:24:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:24:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:24:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-06T08:24:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-04-06T08:24:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ return 0 2024-04-06T08:24:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ status_result=' "state": "Replaying"' 2024-04-06T08:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo ' "state": "Replaying"' 2024-04-06T08:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ grep -i Replaying 2024-04-06T08:24:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: "state": "Replaying" 2024-04-06T08:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=started 2024-04-06T08:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test started = started 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = journal ']' 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' snapshot = snapshot ']' 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_cluster=cluster1 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster2-mirror-image_50.status 2024-04-06T08:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster2-mirror-image_50.status 2024-04-06T08:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local status_log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.status 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ mkfname cluster1-mirror-image_50.status 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ echo cluster1-mirror-image_50.status 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local local_status_log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-04-06T08:24:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Snapshot ID: 217 2024-04-06T08:24:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.status 2024-04-06T08:24:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.status 2024-04-06T08:24:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-04-06T08:24:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-06T08:24:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snapshot_id=217 2024-04-06T08:24:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:24:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.2 2024-04-06T08:24:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-06T08:24:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=117 2024-04-06T08:24:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 217 = 117 2024-04-06T08:24:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.4 2024-04-06T08:24:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-06T08:24:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=117 2024-04-06T08:24:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 217 = 117 2024-04-06T08:24:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.8 2024-04-06T08:24:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-06T08:24:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:24:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:-:1.1: Document is empty 2024-04-06T08:24:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: 2024-04-06T08:24:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:^ 2024-04-06T08:24:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id= 2024-04-06T08:24:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 217 = '' 2024-04-06T08:24:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-06T08:24:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1.6 2024-04-06T08:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local log=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.status 2024-04-06T08:24:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-06T08:24:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-06T08:24:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local primary_snapshot_id=217 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 217 = 217 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:24:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-06T08:24:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 1 2024-04-06T08:24:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_50': 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 128 MiB in 32 objects 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 2 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 12867e194129 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.12867e194129 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:24:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock, non-primary 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:15:42 2024 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:15:42 2024 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:15:42 2024 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: protected: False 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: b626fa42-e982-4b5f-9370-6b8c700fab59 2024-04-06T08:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: false 2024-04-06T08:24:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ compare_image_snaps mirror image_50 snap 2024-04-06T08:24:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:24:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap_name=snap 2024-04-06T08:24:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local ret=0 2024-04-06T08:24:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local rmt_export=/tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.export 2024-04-06T08:24:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local loc_export=/tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.export 2024-04-06T08:24:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.export 2024-04-06T08:24:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.export 2024-04-06T08:24:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:24:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.export 2024-04-06T08:24:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-06T08:24:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cmp /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.export 2024-04-06T08:24:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -f /tmp/tmp.phadx0IwJQ/cluster2-mirror-image_50.export /tmp/tmp.phadx0IwJQ/cluster1-mirror-image_50.export 2024-04-06T08:24:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: delete many images' 2024-04-06T08:24:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T08:24:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T08:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 08:24:19 TEST: delete many images 2024-04-06T08:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 08:24:19 TEST: delete many images 2024-04-06T08:24:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 50 2024-04-06T08:24:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_1 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-04-06T08:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-06T08:24:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:24:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_1 2024-04-06T08:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:24:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-06T08:24:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-06T08:24:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:24:20.810+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55790551eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:24:20.810+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x557904f2e000 [3/eb012d1c-4f5e-44aa-9edc-096c064d4f16] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_2 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-04-06T08:24:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-06T08:24:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:24:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_2 2024-04-06T08:24:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:24:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-06T08:24:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-06T08:24:22.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:22.878+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e243fdba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:22.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:22.878+0000 7fbf78057640 -1 rbd::mirror::ImageReplayer: 0x55e24375d680 [3/594aad0b-58bb-4181-a369-5cf04943564e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_3 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-04-06T08:24:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-06T08:24:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:24:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_3 2024-04-06T08:24:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:24:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-06T08:24:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-06T08:24:24.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:24.882+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e243fdbd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:24.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:24.882+0000 7fbf78057640 -1 rbd::mirror::ImageReplayer: 0x55e24375d400 [3/23b023dd-adbb-4aae-bbd9-c320028b0492] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_4 2024-04-06T08:24:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-04-06T08:24:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:24:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-04-06T08:24:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-06T08:24:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:24:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_4 2024-04-06T08:24:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:24:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-06T08:24:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-06T08:24:27.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:27.081+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e2440f8820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:27.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:27.081+0000 7fbf78057640 -1 rbd::mirror::ImageReplayer: 0x55e244152000 [3/ca09eaed-79f4-4ee1-8861-740c13842056] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_5 2024-04-06T08:24:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-04-06T08:24:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:24:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-04-06T08:24:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-06T08:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_5 2024-04-06T08:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-06T08:24:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-06T08:24:28.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:24:28.889+0000 7f58fdddb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:28.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:24:28.889+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c63680 [3/b950a79f-80c3-4891-a04f-2ec8f28298d1] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:29.774 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T08:24:29.768+0000 7f589d0a0640 -1 reset not still connected to 0x5611920bcf70 2024-04-06T08:24:29.774 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T08:24:29.768+0000 7f589d0a0640 -1 reset not still connected to 0x561194a664e0 2024-04-06T08:24:29.774 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T08:24:29.768+0000 7f589d0a0640 -1 reset not still connected to 0x561196145040 2024-04-06T08:24:29.774 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T08:24:29.768+0000 7f589d0a0640 -1 reset not still connected to 0x56119825d860 2024-04-06T08:24:29.774 INFO:tasks.ceph.cluster1.osd.0.smithi047.stderr:2024-04-06T08:24:29.768+0000 7f589d0a0640 -1 reset not still connected to 0x5611987905b0 2024-04-06T08:24:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_6 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-04-06T08:24:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-06T08:24:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:24:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_6 2024-04-06T08:24:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:24:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-06T08:24:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-06T08:24:30.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:30.921+0000 7f68b9cea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24f520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:30.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:30.921+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955ef00 [3/6347dabe-e24c-40dc-91f2-8040b6675422] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_7 2024-04-06T08:24:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-04-06T08:24:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:24:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-04-06T08:24:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-06T08:24:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:24:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_7 2024-04-06T08:24:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:24:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-06T08:24:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-06T08:24:32.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:32.881+0000 7f68b9cea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24f6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:32.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:32.881+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955ea00 [3/839a8373-291e-4850-b111-46075edd9765] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_8 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-04-06T08:24:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-06T08:24:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:24:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_8 2024-04-06T08:24:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:24:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-06T08:24:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-06T08:24:34.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:24:34.945+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557904d42820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:34.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:24:34.945+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x5579045bcf00 [3/be4f8eb9-e09c-4224-856e-896364fa79d2] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_9 2024-04-06T08:24:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-04-06T08:24:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:24:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:35.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-04-06T08:24:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-06T08:24:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:24:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_9 2024-04-06T08:24:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:24:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-06T08:24:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-06T08:24:36.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:24:36.949+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557904dfd860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:36.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:24:36.949+0000 7fdad2c65640 -1 rbd::mirror::ImageReplayer: 0x5579045bc500 [3/760529a0-2277-4e6f-aa85-bf596b26afe0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_10 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-04-06T08:24:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-06T08:24:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:24:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_10 2024-04-06T08:24:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:24:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-06T08:24:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-06T08:24:38.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:24:38.961+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:38.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:24:38.961+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c62f00 [3/1626ab18-b923-4d9b-b871-b9490fbd81d7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_11 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:24:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-04-06T08:24:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_11 2024-04-06T08:24:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:24:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_11 2024-04-06T08:24:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:24:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-04-06T08:24:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-04-06T08:24:40.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:40.969+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a236680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:40.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:40.969+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe0955e780 [3/f7102142-b3e4-44ed-9508-3bf8555f6258] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_12 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-04-06T08:24:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_12 2024-04-06T08:24:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:24:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_12 2024-04-06T08:24:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:24:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-04-06T08:24:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-04-06T08:24:42.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:42.985+0000 7fbf7184a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e2440f9ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:42.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:42.985+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375ca00 [3/8901ee92-51d2-43ee-985f-683952d0fb7e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_13 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-04-06T08:24:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_13 2024-04-06T08:24:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:24:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_13 2024-04-06T08:24:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:24:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-04-06T08:24:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-04-06T08:24:44.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:44.997+0000 7fbf7184a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e2440f8340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:44.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:44.997+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375c500 [3/580a3f05-2cbb-49eb-bcd9-418388fdf451] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_14 2024-04-06T08:24:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-04-06T08:24:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:24:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-04-06T08:24:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_14 2024-04-06T08:24:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:24:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_14 2024-04-06T08:24:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:24:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-04-06T08:24:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-04-06T08:24:47.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:47.013+0000 7f68b9cea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a236820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:47.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:47.013+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe09a58000 [3/c583fe36-b4d5-459c-9629-9b6a5c24b1c2] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_15 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:24:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-04-06T08:24:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_15 2024-04-06T08:24:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:24:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_15 2024-04-06T08:24:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:24:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-04-06T08:24:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-04-06T08:24:49.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:49.033+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a2369c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:49.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:49.033+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe0955e500 [3/9ab9634b-f0c0-42f0-857e-b5d198938167] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_16 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-04-06T08:24:50.099 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9759971624787057) > weight (0.25) 2024-04-06T08:24:50.099 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4001237475201983) > weight (0.25) 2024-04-06T08:24:50.099 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5928748799550982) > weight (0.25) 2024-04-06T08:24:50.099 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5082270774861832) > weight (0.25) 2024-04-06T08:24:50.100 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-04-06T08:24:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_16 2024-04-06T08:24:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:24:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_16 2024-04-06T08:24:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:24:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-04-06T08:24:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-04-06T08:24:51.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:51.041+0000 7fbf7184a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e244466000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:51.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:51.041+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375cf00 [3/725496c1-43b8-4408-a32f-b66992d7c9ef] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_17 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-04-06T08:24:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_17 2024-04-06T08:24:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:24:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_17 2024-04-06T08:24:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:24:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-04-06T08:24:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-04-06T08:24:53.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:53.053+0000 7f68b9cea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a236b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:53.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:53.053+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe09a58280 [3/ffce73d4-43fe-46e8-9362-ed0490dd6ab0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_18 2024-04-06T08:24:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-04-06T08:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-04-06T08:24:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_18 2024-04-06T08:24:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:24:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_18 2024-04-06T08:24:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:24:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-04-06T08:24:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-04-06T08:24:55.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:24:55.049+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5559177e61a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:55.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:24:55.049+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c63900 [3/2a2e0364-70be-4a9d-8f86-d8dd2c9b39fc] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_19 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-04-06T08:24:56.199 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7944799001419071) > weight (0.25) 2024-04-06T08:24:56.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7791071848509117) > weight (0.25) 2024-04-06T08:24:56.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6164360381777632) > weight (0.25) 2024-04-06T08:24:56.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9153860505052169) > weight (0.25) 2024-04-06T08:24:56.200 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-04-06T08:24:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_19 2024-04-06T08:24:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:24:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_19 2024-04-06T08:24:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:24:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-04-06T08:24:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-04-06T08:24:57.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:57.073+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:57.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:24:57.073+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955f680 [3/73a422d7-74ad-4197-90fc-2e353882f67e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_20 2024-04-06T08:24:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-04-06T08:24:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:24:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-04-06T08:24:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_20 2024-04-06T08:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_20 2024-04-06T08:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-04-06T08:24:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-04-06T08:24:59.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:59.084+0000 7fbf7184a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e2445836c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:24:59.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:24:59.084+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375d180 [3/877e6897-26c6-469a-b566-eb8cf289549d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:24:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:24:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:24:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_21 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:24:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-04-06T08:25:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_21 2024-04-06T08:25:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:25:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_21 2024-04-06T08:25:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:25:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-04-06T08:25:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-04-06T08:25:01.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:01.088+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5579054f49c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:01.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:01.088+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x5579045bd180 [3/a30e9377-aa3c-4d27-9702-2aab3727fd99] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:01.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:01.576+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557905519040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:01.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:01.576+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x5579045bd180 [3/a30e9377-aa3c-4d27-9702-2aab3727fd99] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_22 2024-04-06T08:25:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-04-06T08:25:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:25:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-04-06T08:25:02.769 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T08:25:02.767+0000 7f04e11c1640 -1 reset not still connected to 0x561d689f6a90 2024-04-06T08:25:02.770 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T08:25:02.767+0000 7f04e11c1640 -1 reset not still connected to 0x561d689f7520 2024-04-06T08:25:02.770 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T08:25:02.767+0000 7f04e11c1640 -1 reset not still connected to 0x561d6a877450 2024-04-06T08:25:02.770 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T08:25:02.767+0000 7f04e11c1640 -1 reset not still connected to 0x561d6de74b60 2024-04-06T08:25:02.770 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T08:25:02.767+0000 7f04e11c1640 -1 reset not still connected to 0x561d6fb9cc30 2024-04-06T08:25:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_22 2024-04-06T08:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_22 2024-04-06T08:25:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:25:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-04-06T08:25:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-04-06T08:25:03.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:03.116+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:03.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:03.116+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955f400 [3/ac3a5da1-f39e-465e-a835-c87e8aa02575] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_23 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-04-06T08:25:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_23 2024-04-06T08:25:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:25:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_23 2024-04-06T08:25:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:25:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-04-06T08:25:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-04-06T08:25:05.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:05.112+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5579050ff040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:05.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:05.112+0000 7fdad2c65640 -1 rbd::mirror::ImageReplayer: 0x5579045bc280 [3/ac6d51f9-12c0-4020-8ba6-c41cc112c761] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_24 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:25:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-04-06T08:25:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_24 2024-04-06T08:25:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_24 2024-04-06T08:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:25:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-04-06T08:25:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-04-06T08:25:07.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:07.120+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5579054f5380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:07.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:07.120+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x5579045bc780 [3/8f4562b6-b8d0-438b-a48f-defcceca733b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_25 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-04-06T08:25:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_25 2024-04-06T08:25:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:25:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_25 2024-04-06T08:25:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:25:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-04-06T08:25:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-04-06T08:25:09.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:09.140+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5579054f4000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:09.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:09.140+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x557904f2ef00 [3/f2c3f78a-8f35-4a3c-b4ba-4152fd15fcbe] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_26 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-04-06T08:25:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_26 2024-04-06T08:25:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:25:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_26 2024-04-06T08:25:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:25:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-04-06T08:25:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-04-06T08:25:10.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:10.852+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55790551e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:10.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:10.852+0000 7fdad2c65640 -1 rbd::mirror::ImageReplayer: 0x5579045bd680 [3/e1986487-bd67-45b7-a50c-21436fe14d4d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_27 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:25:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-04-06T08:25:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_27 2024-04-06T08:25:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:25:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_27 2024-04-06T08:25:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:25:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-04-06T08:25:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-04-06T08:25:12.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:12.888+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55790551e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:12.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:12.888+0000 7fdad2c65640 -1 rbd::mirror::ImageReplayer: 0x5579045bca00 [3/8b927270-e88c-47d8-9f3e-375b09dbf2b2] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_28 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-04-06T08:25:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_28 2024-04-06T08:25:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:25:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_28 2024-04-06T08:25:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:25:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-04-06T08:25:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-04-06T08:25:14.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:14.884+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e2442296c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:14.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:14.884+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e244152780 [3/3c43d300-9de9-4d2a-ba10-0d577a7a18c0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_29 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-04-06T08:25:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_29 2024-04-06T08:25:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:25:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_29 2024-04-06T08:25:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:25:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-04-06T08:25:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-04-06T08:25:16.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:16.900+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55790551f040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:16.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:16.900+0000 7fdad2c65640 -1 rbd::mirror::ImageReplayer: 0x5579045bd900 [3/c5b8cfe3-a440-4aa6-b7ed-bbf8e31b2175] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_30 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-04-06T08:25:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_30 2024-04-06T08:25:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:25:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_30 2024-04-06T08:25:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:25:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-04-06T08:25:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-04-06T08:25:18.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:18.920+0000 7f58fdddb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:18.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:18.920+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c63400 [3/0ac3adc2-3f86-4a07-a54a-4fd87c334a97] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_31 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-04-06T08:25:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_31 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_31 2024-04-06T08:25:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:25:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-04-06T08:25:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-04-06T08:25:20.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:20.924+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55790551f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:20.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:20.924+0000 7fdad2c65640 -1 rbd::mirror::ImageReplayer: 0x5579045bcc80 [3/a12859fc-49ad-4646-9f39-4212f7cb141b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_32 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:25:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-04-06T08:25:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_32 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_32 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:22.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:25:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-04-06T08:25:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-04-06T08:25:22.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:22.908+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773f520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:22.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:22.908+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c62280 [3/079afbf7-3e02-4a0c-b385-2d20983af07c] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_33 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:25:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-04-06T08:25:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_33 2024-04-06T08:25:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:25:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_33 2024-04-06T08:25:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:25:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-04-06T08:25:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-04-06T08:25:24.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:24.964+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:24.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:24.964+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c63180 [3/43b614ea-0108-4b77-ae7f-5b1e2003fce5] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_34 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-04-06T08:25:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_34 2024-04-06T08:25:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:25:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_34 2024-04-06T08:25:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:25:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-04-06T08:25:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-04-06T08:25:26.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:26.976+0000 7f68b9cea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:26.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:26.976+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955f900 [3/6d663376-e7fa-4e9d-b581-641ed5f5b057] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:27.003 INFO:tasks.ceph.cluster1.osd.1.smithi047.stderr:2024-04-06T08:25:26.998+0000 7f04e11c1640 -1 reset not still connected to 0x561d6de74b60 2024-04-06T08:25:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_35 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-04-06T08:25:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_35 2024-04-06T08:25:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:25:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_35 2024-04-06T08:25:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:25:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-04-06T08:25:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-04-06T08:25:28.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:28.956+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e24422d1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:28.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:28.956+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375cc80 [3/8219f002-4b2d-40e9-9cd8-ae83bc156dc4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_36 2024-04-06T08:25:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-04-06T08:25:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:25:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-04-06T08:25:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_36 2024-04-06T08:25:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:25:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_36 2024-04-06T08:25:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:25:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-04-06T08:25:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-04-06T08:25:30.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:30.984+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:30.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:30.984+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c62500 [3/111e48f6-921e-4a32-9d71-4f87dd955b5e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_37 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:25:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-04-06T08:25:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_37 2024-04-06T08:25:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:25:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_37 2024-04-06T08:25:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:25:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-04-06T08:25:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-04-06T08:25:33.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:33.235+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:33.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:33.235+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955fb80 [3/9c0cee78-3594-4b38-aea6-9da642e815a7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_38 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:25:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-04-06T08:25:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_38 2024-04-06T08:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_38 2024-04-06T08:25:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:25:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-04-06T08:25:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-04-06T08:25:35.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:35.007+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a237d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:35.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:35.007+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe0a19e500 [3/bee07777-cb98-40aa-a6a6-2b3e5d90cb96] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_39 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-04-06T08:25:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_39 2024-04-06T08:25:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:25:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_39 2024-04-06T08:25:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:25:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-04-06T08:25:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-04-06T08:25:37.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:37.031+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e24422d860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:37.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:37.031+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e244152f00 [3/2f88af95-4f57-486d-93b3-f203c018dd1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_40 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-04-06T08:25:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_40 2024-04-06T08:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_40 2024-04-06T08:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:25:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-04-06T08:25:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-04-06T08:25:39.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:39.051+0000 7f58fdddb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773fa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:39.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:39.051+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c63b80 [3/3436f7ad-064f-4e05-bbf9-791ed79dfc9d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_41 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-04-06T08:25:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_41 2024-04-06T08:25:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:25:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_41 2024-04-06T08:25:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:25:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-04-06T08:25:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-04-06T08:25:41.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:41.055+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555917a81a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:41.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:41.055+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c62780 [3/175e0311-52c5-4f3f-9e73-a36267e32a86] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_42 2024-04-06T08:25:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-04-06T08:25:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:25:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-04-06T08:25:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_42 2024-04-06T08:25:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:25:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_42 2024-04-06T08:25:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:25:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-04-06T08:25:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-04-06T08:25:43.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:43.071+0000 7fdacbc57640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5579054f56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:43.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:43.071+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x5579045bdb80 [3/f200c959-0115-400f-90ae-1785585fdbef] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_43 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-04-06T08:25:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_43 2024-04-06T08:25:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:25:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_43 2024-04-06T08:25:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:25:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-04-06T08:25:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-04-06T08:25:45.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:45.075+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555917518680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:45.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:45.075+0000 7f5904de9640 -1 rbd::mirror::ImageReplayer: 0x555916c62c80 [3/1a6528d8-9d7a-4762-8686-78b7552a3f94] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:45.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:45.347+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773d040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:45.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:45.347+0000 7f59045e8640 -1 rbd::mirror::ImageReplayer: 0x555916c62c80 [3/1a6528d8-9d7a-4762-8686-78b7552a3f94] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_44 2024-04-06T08:25:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-04-06T08:25:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:25:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-04-06T08:25:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_44 2024-04-06T08:25:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:25:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_44 2024-04-06T08:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-04-06T08:25:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-04-06T08:25:47.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:47.079+0000 7f58fe5dc640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:47.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:47.079+0000 7f59045e8640 -1 rbd::mirror::ImageReplayer: 0x5559175f0000 [3/79a01d9a-fb10-4dfa-97e4-0874260e4dde] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_45 2024-04-06T08:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-04-06T08:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-04-06T08:25:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_45 2024-04-06T08:25:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:25:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_45 2024-04-06T08:25:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:25:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-04-06T08:25:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-04-06T08:25:49.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:49.103+0000 7fbf71049640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e24422dd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:49.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:49.103+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375c780 [3/5ba3668d-93d2-4d22-87c3-f63064c87646] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_46 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-04-06T08:25:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_46 2024-04-06T08:25:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:25:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_46 2024-04-06T08:25:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:25:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-04-06T08:25:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-04-06T08:25:51.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:51.119+0000 7fbf7184a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e24422d6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:51.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:25:51.119+0000 7fbf77856640 -1 rbd::mirror::ImageReplayer: 0x55e24375c280 [3/5084cde1-4c2c-469a-b871-05d78b2a8610] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_47 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-04-06T08:25:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_47 2024-04-06T08:25:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:25:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_47 2024-04-06T08:25:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:25:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-04-06T08:25:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-04-06T08:25:53.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:53.143+0000 7f58fdddb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55591773fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:53.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:25:53.143+0000 7f59045e8640 -1 rbd::mirror::ImageReplayer: 0x555916c62a00 [3/339c1583-8140-4531-ae3a-4bb2efc43b9e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_48 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-04-06T08:25:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_48 2024-04-06T08:25:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:25:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_48 2024-04-06T08:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-04-06T08:25:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-04-06T08:25:55.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:55.371+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:55.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:55.375+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0955ec80 [3/ed498893-396c-4438-80fa-19e53e8f4471] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_49 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-04-06T08:25:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_49 2024-04-06T08:25:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:25:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_49 2024-04-06T08:25:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:25:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-04-06T08:25:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-04-06T08:25:57.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:57.143+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a1be1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:57.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:25:57.143+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe09f7aa00 [3/f62960e8-a1e8-461c-bf37-b3f594ac8f8b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_50 2024-04-06T08:25:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-04-06T08:25:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:25:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local snap=snap 2024-04-06T08:25:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-04-06T08:25:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing snap: 100% complete...done. 2024-04-06T08:25:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_50 2024-04-06T08:25:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:25:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:25:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:25:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_50 2024-04-06T08:25:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:25:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:25:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-04-06T08:25:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-04-06T08:25:59.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:59.191+0000 7fdacc458640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5579055181a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:25:59.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:25:59.191+0000 7fdad2464640 -1 rbd::mirror::ImageReplayer: 0x5579045bd400 [3/b626fa42-e982-4b5f-9370-6b8c700fab59] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:25:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-06T08:25:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: image deletions should propagate' 2024-04-06T08:25:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T08:25:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T08:25:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 08:25:59 TEST: image deletions should propagate 2024-04-06T08:25:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 08:25:59 TEST: image deletions should propagate 2024-04-06T08:25:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_count=0 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local count 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local last_count=0 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ true 2024-04-06T08:25:59.836 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 40 2024-04-06T08:25:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:25:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 1 -ne 1 2024-04-06T08:25:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T08:25:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'images: ' 2024-04-06T08:25:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 2 2024-04-06T08:25:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ count=0 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state 2024-04-06T08:25:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 40 2024-04-06T08:25:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in `seq 1 40` 2024-04-06T08:25:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 1 -ne 1 2024-04-06T08:25:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-06T08:25:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep 'image health:' 2024-04-06T08:25:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cut '-d ' -f 3 2024-04-06T08:25:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: failed to query services: (13) Permission denied 2024-04-06T08:25:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ state=OK 2024-04-06T08:25:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test OK = ERROR 2024-04-06T08:25:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test OK = OK 2024-04-06T08:25:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:25:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 50 2024-04-06T08:25:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_1 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:25:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_1 2024-04-06T08:25:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T08:25:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:25:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T08:25:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:25:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:25:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T08:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_1 2024-04-06T08:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T08:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_2 2024-04-06T08:26:00.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_2 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T08:26:00.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T08:26:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-06T08:26:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_2 2024-04-06T08:26:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T08:26:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_3 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_3 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T08:26:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T08:26:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-06T08:26:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T08:26:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_3 2024-04-06T08:26:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T08:26:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T08:26:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_4 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T08:26:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_4 2024-04-06T08:26:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T08:26:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T08:26:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T08:26:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_4 2024-04-06T08:26:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T08:26:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T08:26:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_5 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_5 2024-04-06T08:26:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T08:26:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T08:26:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T08:26:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_5 2024-04-06T08:26:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T08:26:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_6 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-06T08:26:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_6 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T08:26:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T08:26:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-06T08:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T08:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_6 2024-04-06T08:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T08:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T08:26:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_7 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_7 2024-04-06T08:26:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T08:26:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T08:26:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_7 2024-04-06T08:26:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T08:26:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_8 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_8 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T08:26:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T08:26:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T08:26:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_8 2024-04-06T08:26:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T08:26:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_9 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_9 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T08:26:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T08:26:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T08:26:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_9 2024-04-06T08:26:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T08:26:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_10 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_10 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T08:26:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T08:26:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T08:26:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_10 2024-04-06T08:26:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T08:26:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T08:26:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_11 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_11 2024-04-06T08:26:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_11 2024-04-06T08:26:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-06T08:26:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-06T08:26:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_11 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_11 2024-04-06T08:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_11 2024-04-06T08:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-06T08:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-06T08:26:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_12 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_12 2024-04-06T08:26:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_12 2024-04-06T08:26:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-06T08:26:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-06T08:26:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_12 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_12 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_12 2024-04-06T08:26:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-06T08:26:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_13 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_13 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_13 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-06T08:26:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-06T08:26:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_13 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_13 2024-04-06T08:26:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_13 2024-04-06T08:26:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-06T08:26:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-06T08:26:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_14 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_14 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_14 2024-04-06T08:26:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-06T08:26:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-06T08:26:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_14 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_14 2024-04-06T08:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_14 2024-04-06T08:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-06T08:26:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_15 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_15 2024-04-06T08:26:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_15 2024-04-06T08:26:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-06T08:26:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-06T08:26:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_15 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_15 2024-04-06T08:26:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_15 2024-04-06T08:26:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-06T08:26:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-06T08:26:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_16 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_16 2024-04-06T08:26:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_16 2024-04-06T08:26:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-06T08:26:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-06T08:26:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-04-06T08:26:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_16 2024-04-06T08:26:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_16 2024-04-06T08:26:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_16 2024-04-06T08:26:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-06T08:26:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-06T08:26:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_17 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_17 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_17 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-06T08:26:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-06T08:26:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_17 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_17 2024-04-06T08:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_17 2024-04-06T08:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-06T08:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-06T08:26:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_18 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_18 2024-04-06T08:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_18 2024-04-06T08:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-06T08:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-06T08:26:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:03.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_18 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_18 2024-04-06T08:26:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_18 2024-04-06T08:26:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-06T08:26:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_19 2024-04-06T08:26:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_19 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_19 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-06T08:26:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-06T08:26:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_19 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_19 2024-04-06T08:26:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_19 2024-04-06T08:26:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-06T08:26:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_20 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-04-06T08:26:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_20 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_20 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-06T08:26:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-06T08:26:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_20 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_20 2024-04-06T08:26:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_20 2024-04-06T08:26:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-06T08:26:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_21 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_21 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_21 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-06T08:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-06T08:26:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_21 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_21 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_21 2024-04-06T08:26:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-06T08:26:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-06T08:26:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_22 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_22 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_22 2024-04-06T08:26:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-06T08:26:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-06T08:26:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-04-06T08:26:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_22 2024-04-06T08:26:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_22 2024-04-06T08:26:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_22 2024-04-06T08:26:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-06T08:26:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_23 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-04-06T08:26:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_23 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_23 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-06T08:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-06T08:26:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-04-06T08:26:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_23 2024-04-06T08:26:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_23 2024-04-06T08:26:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_23 2024-04-06T08:26:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-06T08:26:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-06T08:26:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_24 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_24 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_24 2024-04-06T08:26:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-06T08:26:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-06T08:26:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_24 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_24 2024-04-06T08:26:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_24 2024-04-06T08:26:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-06T08:26:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_25 2024-04-06T08:26:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_25 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_25 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-06T08:26:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-06T08:26:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_25 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_25 2024-04-06T08:26:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_25 2024-04-06T08:26:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-06T08:26:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-06T08:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_26 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_26 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_26 2024-04-06T08:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-06T08:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-06T08:26:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-04-06T08:26:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_26 2024-04-06T08:26:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_26 2024-04-06T08:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_26 2024-04-06T08:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-06T08:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_27 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_27 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_27 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-06T08:26:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-06T08:26:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-04-06T08:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_27 2024-04-06T08:26:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_27 2024-04-06T08:26:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_27 2024-04-06T08:26:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-06T08:26:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-06T08:26:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_28 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_28 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_28 2024-04-06T08:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-06T08:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-06T08:26:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-04-06T08:26:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_28 2024-04-06T08:26:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_28 2024-04-06T08:26:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_28 2024-04-06T08:26:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-06T08:26:05.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-06T08:26:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_29 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_29 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_29 2024-04-06T08:26:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-06T08:26:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-06T08:26:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_29 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_29 2024-04-06T08:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_29 2024-04-06T08:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-06T08:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_30 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_30 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_30 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-06T08:26:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-06T08:26:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-04-06T08:26:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_30 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_30 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_30 2024-04-06T08:26:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-06T08:26:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_31 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_31 2024-04-06T08:26:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_31 2024-04-06T08:26:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-06T08:26:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-06T08:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-04-06T08:26:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_31 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_31 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_31 2024-04-06T08:26:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-06T08:26:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-06T08:26:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_32 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_32 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_32 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-06T08:26:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-06T08:26:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_32 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_32 2024-04-06T08:26:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_32 2024-04-06T08:26:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-06T08:26:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_33 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_33 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_33 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-06T08:26:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-06T08:26:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-04-06T08:26:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_33 2024-04-06T08:26:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_33 2024-04-06T08:26:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_33 2024-04-06T08:26:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-06T08:26:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-06T08:26:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_34 2024-04-06T08:26:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-04-06T08:26:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:26:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_34 2024-04-06T08:26:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_34 2024-04-06T08:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-06T08:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-06T08:26:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-04-06T08:26:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_34 2024-04-06T08:26:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_34 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_34 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-06T08:26:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-06T08:26:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_35 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_35 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_35 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-06T08:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-06T08:26:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_35 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_35 2024-04-06T08:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_35 2024-04-06T08:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-06T08:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-06T08:26:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_36 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_36 2024-04-06T08:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_36 2024-04-06T08:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-06T08:26:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-06T08:26:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-04-06T08:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_36 2024-04-06T08:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:06.954 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_36 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_36 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-06T08:26:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_37 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:26:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_37 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_37 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-06T08:26:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-06T08:26:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_37 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_37 2024-04-06T08:26:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_37 2024-04-06T08:26:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-06T08:26:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_38 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_38 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_38 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-06T08:26:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-06T08:26:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_38 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:07.345 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_38 2024-04-06T08:26:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_38 2024-04-06T08:26:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-06T08:26:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-06T08:26:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_39 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_39 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_39 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-06T08:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-06T08:26:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-04-06T08:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_39 2024-04-06T08:26:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_39 2024-04-06T08:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_39 2024-04-06T08:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-06T08:26:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-06T08:26:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_40 2024-04-06T08:26:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-04-06T08:26:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:26:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_40 2024-04-06T08:26:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_40 2024-04-06T08:26:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-06T08:26:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.632 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-06T08:26:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-04-06T08:26:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_40 2024-04-06T08:26:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_40 2024-04-06T08:26:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_40 2024-04-06T08:26:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-06T08:26:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_41 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_41 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_41 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-06T08:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-06T08:26:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-04-06T08:26:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_41 2024-04-06T08:26:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:07.960 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_41 2024-04-06T08:26:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_41 2024-04-06T08:26:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-06T08:26:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_42 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_42 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_42 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-06T08:26:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-06T08:26:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-04-06T08:26:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_42 2024-04-06T08:26:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_42 2024-04-06T08:26:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_42 2024-04-06T08:26:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-06T08:26:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_43 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_43 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_43 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-06T08:26:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-06T08:26:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_43 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_43 2024-04-06T08:26:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_43 2024-04-06T08:26:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-06T08:26:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-06T08:26:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_44 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_44 2024-04-06T08:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_44 2024-04-06T08:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-06T08:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-06T08:26:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_44 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_44 2024-04-06T08:26:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_44 2024-04-06T08:26:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-06T08:26:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_45 2024-04-06T08:26:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_45 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_45 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-06T08:26:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-06T08:26:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_45 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_45 2024-04-06T08:26:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_45 2024-04-06T08:26:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-06T08:26:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-06T08:26:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_46 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_46 2024-04-06T08:26:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_46 2024-04-06T08:26:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-06T08:26:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-06T08:26:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_46 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_46 2024-04-06T08:26:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_46 2024-04-06T08:26:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-06T08:26:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_47 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-04-06T08:26:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_47 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_47 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-06T08:26:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-06T08:26:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_47 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_47 2024-04-06T08:26:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_47 2024-04-06T08:26:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-06T08:26:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-06T08:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_48 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_48 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_48 2024-04-06T08:26:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-06T08:26:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-06T08:26:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_48 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_48 2024-04-06T08:26:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_48 2024-04-06T08:26:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-06T08:26:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_49 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-04-06T08:26:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_49 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_49 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-06T08:26:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-06T08:26:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_49 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_49 2024-04-06T08:26:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_49 2024-04-06T08:26:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-06T08:26:09.527 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-06T08:26:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_50 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_50 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_50 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-06T08:26:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-06T08:26:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-04-06T08:26:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_50 2024-04-06T08:26:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_50 2024-04-06T08:26:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_50 2024-04-06T08:26:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-06T08:26:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-06T08:26:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-04-06T08:26:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T08:26:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 08:26:09 TEST: delete images during bootstrap 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 08:26:09 TEST: delete images during bootstrap 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=image 2024-04-06T08:26:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-04-06T08:26:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd: mirroring is already configured for image mode 2024-04-06T08:26:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-04-06T08:26:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=image 2024-04-06T08:26:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-04-06T08:26:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd: mirroring is already configured for image mode 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ start_mirror cluster1 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set_cluster_instance cluster1 cluster instance 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local val=cluster1 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster_var_name=cluster 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local instance_var_name=instance 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cluster=cluster1 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=cluster1 2024-04-06T08:26:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' cluster1 = cluster1 ']' 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ instance=0 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval cluster=cluster1 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ cluster=cluster1 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ eval instance=0 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ instance=0 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=test 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 1 10 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_1 2024-04-06T08:26:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_1 512M 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2024-04-06T08:26:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_1 2024-04-06T08:26:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:26:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-04-06T08:26:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:26:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-06T08:26:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_1': 2024-04-06T08:26:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:26:10.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 191fe8e33ebe 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.191fe8e33ebe 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:26:09 2024 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:26:09 2024 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:26:09 2024 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 7c743766-3b0e-403c-b228-eccd62bfd754 2024-04-06T08:26:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:26:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-06T08:26:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:26:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=5 2024-04-06T08:26:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:26:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:26:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 39557120 bytes 2024-04-06T08:26:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 97229824 bytes 2024-04-06T08:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 158878720 bytes 2024-04-06T08:26:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 219653120 bytes 2024-04-06T08:26:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 225915904 bytes 2024-04-06T08:26:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 278407168 bytes 2024-04-06T08:26:11.945 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T08:26:11.946+0000 7fc4119a2640 -1 reset not still connected to 0x560b37b39040 2024-04-06T08:26:12.148 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 346576896 bytes 2024-04-06T08:26:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 404880384 bytes 2024-04-06T08:26:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 443069440 bytes 2024-04-06T08:26:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 458152960 bytes 2024-04-06T08:26:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 526216192 bytes 2024-04-06T08:26:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 611478528 bytes 2024-04-06T08:26:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 622225408 bytes 2024-04-06T08:26:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 687121408 bytes 2024-04-06T08:26:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 735013888 bytes 2024-04-06T08:26:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 770371584 bytes 2024-04-06T08:26:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 803936256 bytes 2024-04-06T08:26:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 824230912 bytes 2024-04-06T08:26:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 886989824 bytes 2024-04-06T08:26:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 970068992 bytes 2024-04-06T08:26:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 1026926592 bytes 2024-04-06T08:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1114936320 bytes 2024-04-06T08:26:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1188412416 bytes 2024-04-06T08:26:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1211663360 bytes 2024-04-06T08:26:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:26:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:26:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:26:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-04-06T08:26:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_1 2024-04-06T08:26:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T08:26:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b8b885cb6 2024-04-06T08:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_1 present 12b8b885cb6 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b8b885cb6 2024-04-06T08:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T08:26:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_1 2024-04-06T08:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T08:26:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b8b885cb6 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b8b885cb6 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b8b885cb6 -o 12b8b885cb6 = 12b8b885cb6 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_1 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-04-06T08:26:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:26:16.654+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ea2ff400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2024-04-06T08:26:16.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:26:16.654+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ea2ff400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2024-04-06T08:26:19.101 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8778435138593591) > weight (0.25) 2024-04-06T08:26:19.101 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.532840358162983) > weight (0.25) 2024-04-06T08:26:19.101 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3779735785375069) > weight (0.25) 2024-04-06T08:26:19.101 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5860592385344279) > weight (0.25) 2024-04-06T08:26:19.101 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2024-04-06T08:26:19.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:26:19.322+0000 7f68b94e9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe0a24e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:26:19.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:26:19.322+0000 7f68c04f7640 -1 rbd::mirror::ImageReplayer: 0x55fe0a19ea00 [3/7c743766-3b0e-403c-b228-eccd62bfd754] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:26:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:26:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T08:26:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_1 2024-04-06T08:26:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T08:26:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T08:26:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_1 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_1 2024-04-06T08:26:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-06T08:26:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:21.869 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_1 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-04-06T08:26:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-06T08:26:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:26:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:26:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_1 2024-04-06T08:26:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_1 2024-04-06T08:26:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-06T08:26:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-06T08:26:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:26:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_2 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_2 512M 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2024-04-06T08:26:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_2 2024-04-06T08:26:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:26:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-04-06T08:26:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-06T08:26:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_2': 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1937acf98c60 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1937acf98c60 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:26:42 2024 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:26:42 2024 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:26:42 2024 2024-04-06T08:26:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:26:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:26:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 88798935-089e-4c84-84df-93296a1c3cb7 2024-04-06T08:26:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:26:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-06T08:26:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:26:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=7 2024-04-06T08:26:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:26:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 22510592 bytes 2024-04-06T08:26:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 60869632 bytes 2024-04-06T08:26:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 125907968 bytes 2024-04-06T08:26:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 188218368 bytes 2024-04-06T08:26:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 201595904 bytes 2024-04-06T08:26:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 202217472 bytes 2024-04-06T08:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 209210368 bytes 2024-04-06T08:26:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 287697920 bytes 2024-04-06T08:26:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 317813760 bytes 2024-04-06T08:26:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 326974464 bytes 2024-04-06T08:26:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 366094336 bytes 2024-04-06T08:26:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 426174464 bytes 2024-04-06T08:26:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 468428800 bytes 2024-04-06T08:26:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 492422144 bytes 2024-04-06T08:26:45.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 542707712 bytes 2024-04-06T08:26:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 603471872 bytes 2024-04-06T08:26:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 613196800 bytes 2024-04-06T08:26:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 671076352 bytes 2024-04-06T08:26:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 747663360 bytes 2024-04-06T08:26:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 831506432 bytes 2024-04-06T08:26:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 866812928 bytes 2024-04-06T08:26:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 895915008 bytes 2024-04-06T08:26:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 950300672 bytes 2024-04-06T08:26:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 999439360 bytes 2024-04-06T08:26:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1014786048 bytes 2024-04-06T08:26:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1015398400 bytes 2024-04-06T08:26:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1019042816 bytes 2024-04-06T08:26:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1057632256 bytes 2024-04-06T08:26:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1083240448 bytes 2024-04-06T08:26:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1119060992 bytes 2024-04-06T08:26:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1186407424 bytes 2024-04-06T08:26:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1219402752 bytes 2024-04-06T08:26:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1312942080 bytes 2024-04-06T08:26:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1388603392 bytes 2024-04-06T08:26:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1468427264 bytes 2024-04-06T08:26:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1491521536 bytes 2024-04-06T08:26:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1568647168 bytes 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:26:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_2 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T08:26:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b87adba16e 2024-04-06T08:26:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_2 present 12b87adba16e 2024-04-06T08:26:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b87adba16e 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_2 2024-04-06T08:26:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T08:26:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b87adba16e 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b87adba16e 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b87adba16e -o 12b87adba16e = 12b87adba16e 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_2 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-04-06T08:26:51.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:26:51.141+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e9fbf800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2024-04-06T08:26:51.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:26:51.141+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e9fbf800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2024-04-06T08:26:53.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6086636936173854) > weight (0.25) 2024-04-06T08:26:53.201 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8599692054373882) > weight (0.25) 2024-04-06T08:26:53.201 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.28323810351269363) > weight (0.25) 2024-04-06T08:26:53.201 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8064457632886093) > weight (0.25) 2024-04-06T08:26:53.201 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-04-06T08:26:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:26:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T08:26:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_2 2024-04-06T08:26:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T08:26:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T08:26:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:26:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:26:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:26:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-06T08:26:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:26:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:26:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:26:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:26:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:26:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_2 2024-04-06T08:26:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:26:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:26:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_2 2024-04-06T08:26:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-06T08:26:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:26:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_2 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-04-06T08:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-06T08:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_2 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_2 2024-04-06T08:26:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-06T08:26:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-06T08:27:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_3 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_3 512M 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:27:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:27:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:27:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:27:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2024-04-06T08:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_3 2024-04-06T08:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:27:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-04-06T08:27:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:27:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-06T08:27:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_3': 2024-04-06T08:27:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:27:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:27:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:27:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 194fbea12b93 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.194fbea12b93 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:27:19 2024 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:27:19 2024 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:27:19 2024 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 65d50afa-4051-4c38-9af0-4e353c6918f5 2024-04-06T08:27:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:27:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-06T08:27:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:27:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=10 2024-04-06T08:27:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:27:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:27:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 15614976 bytes 2024-04-06T08:27:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 77969408 bytes 2024-04-06T08:27:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 119858176 bytes 2024-04-06T08:27:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 186931200 bytes 2024-04-06T08:27:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 236870656 bytes 2024-04-06T08:27:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 315858944 bytes 2024-04-06T08:27:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 366562304 bytes 2024-04-06T08:27:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 367161344 bytes 2024-04-06T08:27:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 372809728 bytes 2024-04-06T08:27:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 429362176 bytes 2024-04-06T08:27:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 494170112 bytes 2024-04-06T08:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 511347712 bytes 2024-04-06T08:27:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 571503616 bytes 2024-04-06T08:27:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 632416256 bytes 2024-04-06T08:27:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 681318400 bytes 2024-04-06T08:27:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 732016640 bytes 2024-04-06T08:27:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 793465856 bytes 2024-04-06T08:27:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 843026432 bytes 2024-04-06T08:27:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 908147712 bytes 2024-04-06T08:27:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 955829248 bytes 2024-04-06T08:27:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 961559552 bytes 2024-04-06T08:27:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 987630592 bytes 2024-04-06T08:27:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1067057152 bytes 2024-04-06T08:27:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1136084992 bytes 2024-04-06T08:27:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1185563648 bytes 2024-04-06T08:27:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1187850240 bytes 2024-04-06T08:27:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1207960576 bytes 2024-04-06T08:27:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1281093632 bytes 2024-04-06T08:27:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1303708672 bytes 2024-04-06T08:27:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1330263040 bytes 2024-04-06T08:27:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1334718464 bytes 2024-04-06T08:27:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1378256896 bytes 2024-04-06T08:27:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1430432768 bytes 2024-04-06T08:27:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1493248000 bytes 2024-04-06T08:27:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1573404672 bytes 2024-04-06T08:27:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1601302528 bytes 2024-04-06T08:27:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1694733312 bytes 2024-04-06T08:27:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1749523456 bytes 2024-04-06T08:27:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1840493568 bytes 2024-04-06T08:27:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1888620544 bytes 2024-04-06T08:27:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1927678976 bytes 2024-04-06T08:27:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1959204864 bytes 2024-04-06T08:27:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1986540544 bytes 2024-04-06T08:27:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2029468672 bytes 2024-04-06T08:27:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2057926656 bytes 2024-04-06T08:27:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2122236928 bytes 2024-04-06T08:27:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2164477952 bytes 2024-04-06T08:27:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2206300160 bytes 2024-04-06T08:27:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2222221312 bytes 2024-04-06T08:27:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2295665664 bytes 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:27:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_3 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T08:27:30.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:27:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b88fd0bc10 2024-04-06T08:27:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:27:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:27:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_3 present 12b88fd0bc10 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b88fd0bc10 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:27:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:27:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T08:27:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:27:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:27:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_3 2024-04-06T08:27:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T08:27:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b88fd0bc10 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b88fd0bc10 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b88fd0bc10 -o 12b88fd0bc10 = 12b88fd0bc10 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_3 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-04-06T08:27:30.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:27:30.920+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ea5b3c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2024-04-06T08:27:30.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:27:30.920+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ea5b3c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2024-04-06T08:27:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:27:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:27:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T08:27:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:27:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:27:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_3 2024-04-06T08:27:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T08:27:35.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:27:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T08:27:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:27:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:27:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:27:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-06T08:27:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:27:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:27:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:27:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:27:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:27:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_3 2024-04-06T08:27:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:27:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:27:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_3 2024-04-06T08:27:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-06T08:27:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:27:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_3 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-04-06T08:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-06T08:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_3 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_3 2024-04-06T08:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-06T08:27:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-06T08:28:00.102 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3989818503893452) > weight (0.25) 2024-04-06T08:28:00.102 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8674754184031183) > weight (0.25) 2024-04-06T08:28:00.102 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.747316927887099) > weight (0.25) 2024-04-06T08:28:00.103 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8766054403464713) > weight (0.25) 2024-04-06T08:28:00.103 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-04-06T08:28:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:28:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:28:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_4 2024-04-06T08:28:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_4 512M 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2024-04-06T08:28:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_4 2024-04-06T08:28:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:28:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-04-06T08:28:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:28:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-06T08:28:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_4': 2024-04-06T08:28:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:28:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 196784054332 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.196784054332 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:28:01 2024 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:28:01 2024 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:28:01 2024 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8a31a263-0f39-49a2-8b38-96c5a336fc8b 2024-04-06T08:28:01.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:28:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-06T08:28:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:28:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:28:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:28:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:28:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 618496 bytes 2024-04-06T08:28:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 39209984 bytes 2024-04-06T08:28:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 85643264 bytes 2024-04-06T08:28:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 119977984 bytes 2024-04-06T08:28:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 140297216 bytes 2024-04-06T08:28:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 158056448 bytes 2024-04-06T08:28:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 241060864 bytes 2024-04-06T08:28:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 296571904 bytes 2024-04-06T08:28:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 373839872 bytes 2024-04-06T08:28:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 397442048 bytes 2024-04-06T08:28:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 424699904 bytes 2024-04-06T08:28:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 508520448 bytes 2024-04-06T08:28:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 509126656 bytes 2024-04-06T08:28:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 516648960 bytes 2024-04-06T08:28:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 563830784 bytes 2024-04-06T08:28:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 607239168 bytes 2024-04-06T08:28:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 677316608 bytes 2024-04-06T08:28:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 728877056 bytes 2024-04-06T08:28:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 754287616 bytes 2024-04-06T08:28:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 778791936 bytes 2024-04-06T08:28:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 821256192 bytes 2024-04-06T08:28:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 894792704 bytes 2024-04-06T08:28:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 987255808 bytes 2024-04-06T08:28:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1010633728 bytes 2024-04-06T08:28:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1105807360 bytes 2024-04-06T08:28:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1106408448 bytes 2024-04-06T08:28:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1140357120 bytes 2024-04-06T08:28:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1226226688 bytes 2024-04-06T08:28:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1271280640 bytes 2024-04-06T08:28:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1360215040 bytes 2024-04-06T08:28:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1413575680 bytes 2024-04-06T08:28:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1424498688 bytes 2024-04-06T08:28:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1436393472 bytes 2024-04-06T08:28:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1495782400 bytes 2024-04-06T08:28:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1516316672 bytes 2024-04-06T08:28:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1516939264 bytes 2024-04-06T08:28:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1531195392 bytes 2024-04-06T08:28:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1576253440 bytes 2024-04-06T08:28:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1609678848 bytes 2024-04-06T08:28:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1642606592 bytes 2024-04-06T08:28:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1754974208 bytes 2024-04-06T08:28:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1807182848 bytes 2024-04-06T08:28:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1846682624 bytes 2024-04-06T08:28:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1863002112 bytes 2024-04-06T08:28:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1908255744 bytes 2024-04-06T08:28:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1957793792 bytes 2024-04-06T08:28:10.473 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2021306368 bytes 2024-04-06T08:28:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2063912960 bytes 2024-04-06T08:28:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2148518912 bytes 2024-04-06T08:28:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2198419456 bytes 2024-04-06T08:28:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2221036544 bytes 2024-04-06T08:28:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2252444672 bytes 2024-04-06T08:28:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2272553984 bytes 2024-04-06T08:28:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2285785088 bytes 2024-04-06T08:28:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2321034240 bytes 2024-04-06T08:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2367902720 bytes 2024-04-06T08:28:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2406156288 bytes 2024-04-06T08:28:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2465736704 bytes 2024-04-06T08:28:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2493078528 bytes 2024-04-06T08:28:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2513373184 bytes 2024-04-06T08:28:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2617798656 bytes 2024-04-06T08:28:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2681368576 bytes 2024-04-06T08:28:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2687609856 bytes 2024-04-06T08:28:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2745149440 bytes 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:28:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_4 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T08:28:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:28:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b8c1cbb530 2024-04-06T08:28:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:28:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_4 present 12b8c1cbb530 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b8c1cbb530 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:28:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T08:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_4 2024-04-06T08:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T08:28:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b8c1cbb530 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b8c1cbb530 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b8c1cbb530 -o 12b8c1cbb530 = 12b8c1cbb530 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_4 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-04-06T08:28:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:28:14.215+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e94d9800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2024-04-06T08:28:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:28:14.215+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e94d9800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2024-04-06T08:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:28:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_4 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T08:28:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:28:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T08:28:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:28:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:28:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:28:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-06T08:28:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:28:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:28:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:28:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:28:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:28:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_4 2024-04-06T08:28:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:28:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:28:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_4 2024-04-06T08:28:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-06T08:28:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:28:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-06T08:28:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:28:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:28:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:28:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_4 2024-04-06T08:28:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-04-06T08:28:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-06T08:28:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:28:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:28:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_4 2024-04-06T08:28:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_4 2024-04-06T08:28:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-06T08:28:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-06T08:28:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:28:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:28:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_5 2024-04-06T08:28:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_5 512M 2024-04-06T08:28:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:28:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2024-04-06T08:28:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_5 2024-04-06T08:28:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:28:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:28:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-04-06T08:28:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:28:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-06T08:28:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_5': 2024-04-06T08:28:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:28:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:28:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:28:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 197fac054dc7 2024-04-06T08:28:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.197fac054dc7 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:28:44 2024 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:28:44 2024 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:28:44 2024 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: fc829ac8-e25a-4a80-85ca-4d1f39b704e5 2024-04-06T08:28:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:28:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-06T08:28:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:28:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:28:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:28:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:28:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:28:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 28370944 bytes 2024-04-06T08:28:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 104242176 bytes 2024-04-06T08:28:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 138158080 bytes 2024-04-06T08:28:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 186804224 bytes 2024-04-06T08:28:46.202 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T08:28:46.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T08:28:46.202 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T08:28:46.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:28:46.202+0000 7f59055ea640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 52239) UID: 1000 2024-04-06T08:28:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 212344832 bytes 2024-04-06T08:28:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 227796992 bytes 2024-04-06T08:28:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 258463744 bytes 2024-04-06T08:28:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 295628800 bytes 2024-04-06T08:28:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 344005632 bytes 2024-04-06T08:28:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 360203264 bytes 2024-04-06T08:28:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 452655104 bytes 2024-04-06T08:28:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 464126976 bytes 2024-04-06T08:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 517673984 bytes 2024-04-06T08:28:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 556592128 bytes 2024-04-06T08:28:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 588548096 bytes 2024-04-06T08:28:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 597754880 bytes 2024-04-06T08:28:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 665200640 bytes 2024-04-06T08:28:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 768897024 bytes 2024-04-06T08:28:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 813970432 bytes 2024-04-06T08:28:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 895238144 bytes 2024-04-06T08:28:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 925684736 bytes 2024-04-06T08:28:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 971375616 bytes 2024-04-06T08:28:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 992949248 bytes 2024-04-06T08:28:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1107147776 bytes 2024-04-06T08:28:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1136369664 bytes 2024-04-06T08:28:50.203 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T08:28:50.204 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:28:50.204 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T08:28:50.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T08:28:50.205 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T08:28:50.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T08:28:50.207 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-04-06T08:28:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1174215680 bytes 2024-04-06T08:28:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1240488960 bytes 2024-04-06T08:28:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1271896064 bytes 2024-04-06T08:28:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1310650368 bytes 2024-04-06T08:28:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1311250432 bytes 2024-04-06T08:28:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1311881216 bytes 2024-04-06T08:28:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1332204544 bytes 2024-04-06T08:28:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1369164800 bytes 2024-04-06T08:28:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1386400768 bytes 2024-04-06T08:28:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1402023936 bytes 2024-04-06T08:28:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1479629824 bytes 2024-04-06T08:28:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1523475456 bytes 2024-04-06T08:28:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1563450368 bytes 2024-04-06T08:28:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1623885824 bytes 2024-04-06T08:28:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1658545152 bytes 2024-04-06T08:28:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1717123072 bytes 2024-04-06T08:28:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1750254592 bytes 2024-04-06T08:28:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1750880256 bytes 2024-04-06T08:28:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1830373376 bytes 2024-04-06T08:28:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1907630080 bytes 2024-04-06T08:28:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1917893632 bytes 2024-04-06T08:28:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1951623168 bytes 2024-04-06T08:28:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1976266752 bytes 2024-04-06T08:28:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2005020672 bytes 2024-04-06T08:28:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2062735360 bytes 2024-04-06T08:28:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2133587968 bytes 2024-04-06T08:28:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2223626240 bytes 2024-04-06T08:28:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2281377792 bytes 2024-04-06T08:28:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2314146816 bytes 2024-04-06T08:28:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2379699200 bytes 2024-04-06T08:28:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2415575040 bytes 2024-04-06T08:28:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2456738816 bytes 2024-04-06T08:28:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2457322496 bytes 2024-04-06T08:28:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2545902592 bytes 2024-04-06T08:28:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2630448128 bytes 2024-04-06T08:28:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2643112960 bytes 2024-04-06T08:28:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2676156416 bytes 2024-04-06T08:28:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2743758848 bytes 2024-04-06T08:28:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2775284736 bytes 2024-04-06T08:28:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2779996160 bytes 2024-04-06T08:28:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2780611584 bytes 2024-04-06T08:28:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2788350976 bytes 2024-04-06T08:28:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2855330816 bytes 2024-04-06T08:28:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2880874496 bytes 2024-04-06T08:28:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2916133888 bytes 2024-04-06T08:28:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2996705280 bytes 2024-04-06T08:28:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2997299200 bytes 2024-04-06T08:28:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 3026537472 bytes 2024-04-06T08:28:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 3105241088 bytes 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:28:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T08:28:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:28:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:28:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_5 2024-04-06T08:28:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T08:28:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:28:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b877321c6f 2024-04-06T08:28:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:28:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_5 present 12b877321c6f 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b877321c6f 2024-04-06T08:28:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:28:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:28:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T08:28:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:28:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:28:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_5 2024-04-06T08:28:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T08:28:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b877321c6f 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b877321c6f 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b877321c6f -o 12b877321c6f = 12b877321c6f 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_5 2024-04-06T08:28:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:28:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:28:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:28:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-04-06T08:29:00.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:29:00.014+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e95a9800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2024-04-06T08:29:00.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:29:00.014+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e95a9800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2024-04-06T08:29:03.104 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9406351139674387) > weight (0.25) 2024-04-06T08:29:03.104 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2734259698421321) > weight (0.25) 2024-04-06T08:29:03.104 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T08:29:03.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T08:29:03.105 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-06T08:29:03.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:29:03.106+0000 7f92dffe2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 45118) UID: 1000 2024-04-06T08:29:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:29:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:29:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T08:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_5 2024-04-06T08:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T08:29:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:29:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T08:29:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:29:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:29:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:29:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:29:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_5 2024-04-06T08:29:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:29:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:29:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_5 2024-04-06T08:29:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-06T08:29:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-06T08:29:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:29:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_5 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:29:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-04-06T08:29:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-06T08:29:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:29:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:29:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:29:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_5 2024-04-06T08:29:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_5 2024-04-06T08:29:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-06T08:29:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-06T08:29:07.106 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T08:29:07.106 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:29:07.107 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T08:29:07.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T08:29:07.107 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T08:29:07.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T08:29:07.111 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-04-06T08:29:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_6 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_6 512M 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:29:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2024-04-06T08:29:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_6 2024-04-06T08:29:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:29:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-04-06T08:29:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:29:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-06T08:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_6': 2024-04-06T08:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 19a6be0e6209 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.19a6be0e6209 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:29:28 2024 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:29:28 2024 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:29:28 2024 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: facf531c-584a-4e27-ace4-5655fb613193 2024-04-06T08:29:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-06T08:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:29:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=8 2024-04-06T08:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:29:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 50933760 bytes 2024-04-06T08:29:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 97760256 bytes 2024-04-06T08:29:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 115827712 bytes 2024-04-06T08:29:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 128044032 bytes 2024-04-06T08:29:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 143890432 bytes 2024-04-06T08:29:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 166892544 bytes 2024-04-06T08:29:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 195954688 bytes 2024-04-06T08:29:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 238967808 bytes 2024-04-06T08:29:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 294902784 bytes 2024-04-06T08:29:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 295534592 bytes 2024-04-06T08:29:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 311020544 bytes 2024-04-06T08:29:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 352072704 bytes 2024-04-06T08:29:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 388902912 bytes 2024-04-06T08:29:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 409694208 bytes 2024-04-06T08:29:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 465269760 bytes 2024-04-06T08:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 515410944 bytes 2024-04-06T08:29:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 532251648 bytes 2024-04-06T08:29:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 544765952 bytes 2024-04-06T08:29:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 557342720 bytes 2024-04-06T08:29:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 568537088 bytes 2024-04-06T08:29:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 579751936 bytes 2024-04-06T08:29:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 624728064 bytes 2024-04-06T08:29:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 642983936 bytes 2024-04-06T08:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 690336768 bytes 2024-04-06T08:29:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 707396608 bytes 2024-04-06T08:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 714371072 bytes 2024-04-06T08:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 767694848 bytes 2024-04-06T08:29:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 793266176 bytes 2024-04-06T08:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 858384384 bytes 2024-04-06T08:29:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 891731968 bytes 2024-04-06T08:29:33.898 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1009119232 bytes 2024-04-06T08:29:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1059927040 bytes 2024-04-06T08:29:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1115160576 bytes 2024-04-06T08:29:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1161955328 bytes 2024-04-06T08:29:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1202324480 bytes 2024-04-06T08:29:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1258889216 bytes 2024-04-06T08:29:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1302423552 bytes 2024-04-06T08:29:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1334521856 bytes 2024-04-06T08:29:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1335141376 bytes 2024-04-06T08:29:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1406093312 bytes 2024-04-06T08:29:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1488422912 bytes 2024-04-06T08:29:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1528265728 bytes 2024-04-06T08:29:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1609746432 bytes 2024-04-06T08:29:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1689637888 bytes 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_6 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:29:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T08:29:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b8e9c81188 2024-04-06T08:29:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:29:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:29:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_6 present 12b8e9c81188 2024-04-06T08:29:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:29:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b8e9c81188 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_6 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T08:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:29:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b8e9c81188 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b8e9c81188 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b8e9c81188 -o 12b8e9c81188 = 12b8e9c81188 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_6 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-04-06T08:29:37.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:29:37.189+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e9812000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2024-04-06T08:29:37.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:29:37.189+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e9812000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=388, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2024-04-06T08:29:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:29:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-06T08:29:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:29:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:29:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:29:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:29:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:29:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T08:29:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:29:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:29:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_6 2024-04-06T08:29:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T08:29:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:29:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T08:29:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:29:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:29:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:29:42.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:29:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_6 2024-04-06T08:29:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:29:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:29:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_6 2024-04-06T08:29:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-06T08:29:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:29:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_6 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-04-06T08:29:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-06T08:29:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:29:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:29:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_6 2024-04-06T08:29:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:29:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:29:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_6 2024-04-06T08:29:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-06T08:29:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-06T08:30:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:30:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_7 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_7 512M 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:30:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:30:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:30:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2024-04-06T08:30:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_7 2024-04-06T08:30:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:30:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-04-06T08:30:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:30:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-06T08:30:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_7': 2024-04-06T08:30:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:30:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:30:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:30:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 19b99986c162 2024-04-06T08:30:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.19b99986c162 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:30:07 2024 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:30:07 2024 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:30:07 2024 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 3e4a2a1d-6fbb-44f0-aa7d-9795475c83ee 2024-04-06T08:30:07.827 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:30:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-06T08:30:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:30:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=12 2024-04-06T08:30:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:30:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:30:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 33398784 bytes 2024-04-06T08:30:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 59886592 bytes 2024-04-06T08:30:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 85401600 bytes 2024-04-06T08:30:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 112128000 bytes 2024-04-06T08:30:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 169164800 bytes 2024-04-06T08:30:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 185900032 bytes 2024-04-06T08:30:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 245359616 bytes 2024-04-06T08:30:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 296800256 bytes 2024-04-06T08:30:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 350704640 bytes 2024-04-06T08:30:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 378805248 bytes 2024-04-06T08:30:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 403635200 bytes 2024-04-06T08:30:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 416425984 bytes 2024-04-06T08:30:10.101 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 469832704 bytes 2024-04-06T08:30:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 495767552 bytes 2024-04-06T08:30:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 542796800 bytes 2024-04-06T08:30:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 592921600 bytes 2024-04-06T08:30:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 693677056 bytes 2024-04-06T08:30:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 707564544 bytes 2024-04-06T08:30:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 709217280 bytes 2024-04-06T08:30:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 773139456 bytes 2024-04-06T08:30:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 816991232 bytes 2024-04-06T08:30:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 818315264 bytes 2024-04-06T08:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 825302016 bytes 2024-04-06T08:30:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 825913344 bytes 2024-04-06T08:30:12.038 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 844398592 bytes 2024-04-06T08:30:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 882716672 bytes 2024-04-06T08:30:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 896087040 bytes 2024-04-06T08:30:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 944236544 bytes 2024-04-06T08:30:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 984332288 bytes 2024-04-06T08:30:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1020027904 bytes 2024-04-06T08:30:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1062940672 bytes 2024-04-06T08:30:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1087746048 bytes 2024-04-06T08:30:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1137905664 bytes 2024-04-06T08:30:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1194372096 bytes 2024-04-06T08:30:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1203273728 bytes 2024-04-06T08:30:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1233393664 bytes 2024-04-06T08:30:14.111 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8932245328124033) > weight (0.25) 2024-04-06T08:30:14.112 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.43375717234123135) > weight (0.25) 2024-04-06T08:30:14.112 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-06T08:30:14.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-06T08:30:14.112 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T08:30:14.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:30:14.112+0000 7f50fd455640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 63156) UID: 1000 2024-04-06T08:30:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1234013184 bytes 2024-04-06T08:30:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1268354048 bytes 2024-04-06T08:30:14.395 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T08:30:14.392+0000 7fc41557d640 -1 bdev(0x560b3110ee00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc41557d640 time 2024-04-06T08:30:14.397986+0000 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560b2dab3bb6] 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.396 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.397 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:*** Caught signal (Aborted) ** 2024-04-06T08:30:14.397 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: in thread 7fc41557d640 thread_name:bstore_aio 2024-04-06T08:30:14.397 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T08:30:14.396+0000 7fc41557d640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc41557d640 time 2024-04-06T08:30:14.397986+0000 2024-04-06T08:30:14.397 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-06T08:30:14.397 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.397 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560b2dab3bb6] 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc422b6f520] 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: pthread_kill() 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: raise() 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: abort() 2024-04-06T08:30:14.398 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560b2dab3c6b] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 6: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 7: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:2024-04-06T08:30:14.396+0000 7fc41557d640 -1 *** Caught signal (Aborted) ** 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: in thread 7fc41557d640 thread_name:bstore_aio 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc422b6f520] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: pthread_kill() 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: raise() 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: abort() 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560b2dab3c6b] 2024-04-06T08:30:14.399 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 6: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.400 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 7: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.400 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.400 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.400 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-06T08:30:14.400 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.421 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: -418> 2024-04-06T08:30:14.392+0000 7fc41557d640 -1 bdev(0x560b3110ee00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-04-06T08:30:14.421 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: -359> 2024-04-06T08:30:14.396+0000 7fc41557d640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc41557d640 time 2024-04-06T08:30:14.397986+0000 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560b2dab3bb6] 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: -39> 2024-04-06T08:30:14.396+0000 7fc41557d640 -1 *** Caught signal (Aborted) ** 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: in thread 7fc41557d640 thread_name:bstore_aio 2024-04-06T08:30:14.422 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc422b6f520] 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: pthread_kill() 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: raise() 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: abort() 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560b2dab3c6b] 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 6: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 7: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-06T08:30:14.423 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: -1419> 2024-04-06T08:30:14.392+0000 7fc41557d640 -1 bdev(0x560b3110ee00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: -1360> 2024-04-06T08:30:14.396+0000 7fc41557d640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc41557d640 time 2024-04-06T08:30:14.397986+0000 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560b2dab3bb6] 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.433 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: -1040> 2024-04-06T08:30:14.396+0000 7fc41557d640 -1 *** Caught signal (Aborted) ** 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: in thread 7fc41557d640 thread_name:bstore_aio 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable) 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc422b6f520] 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2: pthread_kill() 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 3: raise() 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 4: abort() 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560b2dab3c6b] 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 6: (KernelDevice::_aio_thread()+0x9e2) [0x560b2e41cb82] 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 7: ceph-osd(+0xd36451) [0x560b2e41d451] 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc422bc1b43] 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc422c53a00] 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-06T08:30:14.434 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr: 2024-04-06T08:30:14.593 INFO:tasks.ceph.cluster2.osd.0.smithi129.stderr:daemon-helper: command failed with exit status 5 2024-04-06T08:30:15.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-04-06T08:30:16.113 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-06T08:30:16.113 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:30:16.114 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-06T08:30:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-06T08:30:16.114 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-06T08:30:16.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-06T08:30:16.117 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-04-06T08:30:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1355771904 bytes 2024-04-06T08:30:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1431161856 bytes 2024-04-06T08:30:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1446130688 bytes 2024-04-06T08:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1475988480 bytes 2024-04-06T08:30:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1492337664 bytes 2024-04-06T08:30:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1535419392 bytes 2024-04-06T08:30:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1608060928 bytes 2024-04-06T08:30:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1633826816 bytes 2024-04-06T08:30:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1649803264 bytes 2024-04-06T08:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 1734642688 bytes 2024-04-06T08:30:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 1777481728 bytes 2024-04-06T08:30:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 1841953792 bytes 2024-04-06T08:30:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 1876277248 bytes 2024-04-06T08:30:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 1931693056 bytes 2024-04-06T08:30:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 1995059200 bytes 2024-04-06T08:30:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2030086144 bytes 2024-04-06T08:30:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2032419840 bytes 2024-04-06T08:30:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2078323712 bytes 2024-04-06T08:30:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2135590912 bytes 2024-04-06T08:30:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2209572864 bytes 2024-04-06T08:30:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2279747584 bytes 2024-04-06T08:30:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2312600576 bytes 2024-04-06T08:30:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2347719680 bytes 2024-04-06T08:30:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2390211584 bytes 2024-04-06T08:30:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2430665728 bytes 2024-04-06T08:30:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2458182656 bytes 2024-04-06T08:30:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2481019904 bytes 2024-04-06T08:30:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T08:30:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:30:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:30:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_7 2024-04-06T08:30:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T08:30:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:30:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b8fe51506a 2024-04-06T08:30:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:30:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:30:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_7 present 12b8fe51506a 2024-04-06T08:30:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:30:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:30:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b8fe51506a 2024-04-06T08:30:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:30:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:30:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T08:30:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:30:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:30:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_7 2024-04-06T08:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T08:30:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:30:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b8fe51506a 2024-04-06T08:30:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b8fe51506a 2024-04-06T08:30:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b8fe51506a -o 12b8fe51506a = 12b8fe51506a 2024-04-06T08:30:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_7 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-04-06T08:30:20.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-04-06T08:30:20.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:30:20.720+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ea3c0c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2024-04-06T08:30:20.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:30:20.720+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ea3c0c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=390, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2024-04-06T08:30:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:30:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:30:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:30:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T08:30:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:30:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:30:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_7 2024-04-06T08:30:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T08:30:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:30:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T08:30:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:30:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:30:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:30:25.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2024-04-06T08:30:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:30:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_7 2024-04-06T08:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_7 2024-04-06T08:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-06T08:30:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:30:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-06T08:30:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:30:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:30:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:30:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_7 2024-04-06T08:30:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-04-06T08:30:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-06T08:30:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:30:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:30:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_7 2024-04-06T08:30:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_7 2024-04-06T08:30:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-06T08:30:26.270 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-06T08:30:31.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2024-04-06T08:30:36.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2024-04-06T08:30:37.208 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-06T08:30:37.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-06T08:30:37.209 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-06T08:30:37.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:30:37.212+0000 7f46f82de640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 62842) UID: 1000 2024-04-06T08:30:40.209 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-06T08:30:40.209 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:30:40.210 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-06T08:30:40.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-06T08:30:40.210 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-06T08:30:40.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-06T08:30:40.213 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-04-06T08:30:42.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2024-04-06T08:30:47.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2024-04-06T08:30:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_8 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_8 512M 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:30:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2024-04-06T08:30:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_8 2024-04-06T08:30:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:30:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:30:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-04-06T08:30:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:30:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_8': 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 19dc73e0c14 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.19dc73e0c14 2024-04-06T08:30:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:30:50 2024 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:30:50 2024 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:30:50 2024 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 59f2f9ed-0d4e-4881-a142-125a777e9f04 2024-04-06T08:30:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:30:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-06T08:30:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:30:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:30:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=8 2024-04-06T08:30:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:30:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:30:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 24274944 bytes 2024-04-06T08:30:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 35530752 bytes 2024-04-06T08:30:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 36132864 bytes 2024-04-06T08:30:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 126465024 bytes 2024-04-06T08:30:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 214757376 bytes 2024-04-06T08:30:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 265247744 bytes 2024-04-06T08:30:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 281745408 bytes 2024-04-06T08:30:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 326682624 bytes 2024-04-06T08:30:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 422017024 bytes 2024-04-06T08:30:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 445090816 bytes 2024-04-06T08:30:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 501971968 bytes 2024-04-06T08:30:52.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2024-04-06T08:30:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 543795200 bytes 2024-04-06T08:30:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 585063424 bytes 2024-04-06T08:30:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 626877440 bytes 2024-04-06T08:30:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 676490240 bytes 2024-04-06T08:30:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 706324480 bytes 2024-04-06T08:30:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 741639168 bytes 2024-04-06T08:30:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 785217536 bytes 2024-04-06T08:30:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 837821440 bytes 2024-04-06T08:30:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 859877376 bytes 2024-04-06T08:30:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 884930560 bytes 2024-04-06T08:30:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 901033984 bytes 2024-04-06T08:30:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 926875648 bytes 2024-04-06T08:30:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 991521792 bytes 2024-04-06T08:30:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1028566016 bytes 2024-04-06T08:30:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1080816640 bytes 2024-04-06T08:30:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1163762688 bytes 2024-04-06T08:30:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1249370112 bytes 2024-04-06T08:30:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1270484992 bytes 2024-04-06T08:30:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1275698176 bytes 2024-04-06T08:30:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1322262528 bytes 2024-04-06T08:30:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1361909760 bytes 2024-04-06T08:30:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1403196416 bytes 2024-04-06T08:30:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1414262784 bytes 2024-04-06T08:30:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1417320448 bytes 2024-04-06T08:30:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1521398784 bytes 2024-04-06T08:30:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1552025600 bytes 2024-04-06T08:30:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1637197824 bytes 2024-04-06T08:30:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1684227072 bytes 2024-04-06T08:30:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1777295360 bytes 2024-04-06T08:30:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1809210368 bytes 2024-04-06T08:30:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1857485824 bytes 2024-04-06T08:30:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1902566400 bytes 2024-04-06T08:30:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1958495232 bytes 2024-04-06T08:30:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2006374400 bytes 2024-04-06T08:30:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2041503744 bytes 2024-04-06T08:30:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2094402560 bytes 2024-04-06T08:30:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2172130304 bytes 2024-04-06T08:30:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2209785856 bytes 2024-04-06T08:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2275919872 bytes 2024-04-06T08:30:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2324811776 bytes 2024-04-06T08:30:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2404349952 bytes 2024-04-06T08:30:58.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2024-04-06T08:30:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2470975488 bytes 2024-04-06T08:30:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2509688832 bytes 2024-04-06T08:30:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2535559168 bytes 2024-04-06T08:30:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2599261184 bytes 2024-04-06T08:30:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2602824704 bytes 2024-04-06T08:30:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2675651584 bytes 2024-04-06T08:30:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2720976896 bytes 2024-04-06T08:30:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2751904768 bytes 2024-04-06T08:30:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2798185472 bytes 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:30:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_8 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T08:30:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:30:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b8bb45aee3 2024-04-06T08:30:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:30:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:30:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_8 present 12b8bb45aee3 2024-04-06T08:30:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:30:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b8bb45aee3 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:30:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_8 2024-04-06T08:30:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T08:30:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b8bb45aee3 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b8bb45aee3 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b8bb45aee3 -o 12b8bb45aee3 = 12b8bb45aee3 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_8 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:30:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-04-06T08:31:00.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:31:00.031+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581eb858800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2024-04-06T08:31:00.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:31:00.031+0000 7f744c875640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581eb858800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=392, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2024-04-06T08:31:03.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2024-04-06T08:31:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:31:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-06T08:31:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:31:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:31:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:31:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:31:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:31:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:31:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T08:31:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:31:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:31:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_8 2024-04-06T08:31:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T08:31:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:31:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T08:31:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:31:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:31:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:31:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-06T08:31:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:31:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:31:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:31:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:31:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:31:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:31:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_8 2024-04-06T08:31:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:31:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:31:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_8 2024-04-06T08:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-06T08:31:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-06T08:31:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:31:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:31:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:31:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:31:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_8 2024-04-06T08:31:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:31:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-04-06T08:31:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-06T08:31:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:31:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:31:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:31:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_8 2024-04-06T08:31:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_8 2024-04-06T08:31:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-06T08:31:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-06T08:31:09.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2024-04-06T08:31:14.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2024-04-06T08:31:19.859 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2024-04-06T08:31:25.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2024-04-06T08:31:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:31:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:31:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_9 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_9 512M 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:31:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:31:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:31:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:31:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:31:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2024-04-06T08:31:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_9 2024-04-06T08:31:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:31:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-04-06T08:31:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:31:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-06T08:31:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_9': 2024-04-06T08:31:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:31:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:31:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:31:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 19f41a617f99 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.19f41a617f99 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:31:29 2024 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:31:29 2024 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:31:29 2024 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 1b87c85d-b1ec-409c-8b37-9e8041236c73 2024-04-06T08:31:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:31:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-06T08:31:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:31:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=13 2024-04-06T08:31:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:31:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:31:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 28918784 bytes 2024-04-06T08:31:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 43674624 bytes 2024-04-06T08:31:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 136308736 bytes 2024-04-06T08:31:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 212220928 bytes 2024-04-06T08:31:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 223828992 bytes 2024-04-06T08:31:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 265905152 bytes 2024-04-06T08:31:30.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2024-04-06T08:31:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 331134976 bytes 2024-04-06T08:31:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 393748480 bytes 2024-04-06T08:31:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 415634432 bytes 2024-04-06T08:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 416266240 bytes 2024-04-06T08:31:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 441204736 bytes 2024-04-06T08:31:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 549860352 bytes 2024-04-06T08:31:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 575243264 bytes 2024-04-06T08:31:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 623528960 bytes 2024-04-06T08:31:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 667645952 bytes 2024-04-06T08:31:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 704051200 bytes 2024-04-06T08:31:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 741291008 bytes 2024-04-06T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 789975040 bytes 2024-04-06T08:31:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 851288064 bytes 2024-04-06T08:31:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 890318848 bytes 2024-04-06T08:31:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 950325248 bytes 2024-04-06T08:31:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 1011985408 bytes 2024-04-06T08:31:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 1112867840 bytes 2024-04-06T08:31:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1200228352 bytes 2024-04-06T08:31:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1250069504 bytes 2024-04-06T08:31:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1278954496 bytes 2024-04-06T08:31:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1321433088 bytes 2024-04-06T08:31:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1335629824 bytes 2024-04-06T08:31:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1355218944 bytes 2024-04-06T08:31:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1427914752 bytes 2024-04-06T08:31:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1449046016 bytes 2024-04-06T08:31:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1508151296 bytes 2024-04-06T08:31:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1557799936 bytes 2024-04-06T08:31:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1583136768 bytes 2024-04-06T08:31:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1641362432 bytes 2024-04-06T08:31:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1678465024 bytes 2024-04-06T08:31:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1722018816 bytes 2024-04-06T08:31:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1771899904 bytes 2024-04-06T08:31:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1809176576 bytes 2024-04-06T08:31:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1844289536 bytes 2024-04-06T08:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1923014656 bytes 2024-04-06T08:31:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1985369088 bytes 2024-04-06T08:31:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 2045003776 bytes 2024-04-06T08:31:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 2068791296 bytes 2024-04-06T08:31:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 2109558784 bytes 2024-04-06T08:31:35.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 2127197184 bytes 2024-04-06T08:31:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 2167983104 bytes 2024-04-06T08:31:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2208586752 bytes 2024-04-06T08:31:36.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2024-04-06T08:31:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2228393984 bytes 2024-04-06T08:31:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2266647552 bytes 2024-04-06T08:31:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2343139328 bytes 2024-04-06T08:31:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2428041216 bytes 2024-04-06T08:31:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2462094336 bytes 2024-04-06T08:31:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2558839808 bytes 2024-04-06T08:31:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2590093312 bytes 2024-04-06T08:31:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2623870976 bytes 2024-04-06T08:31:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2668951552 bytes 2024-04-06T08:31:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2690362368 bytes 2024-04-06T08:31:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2698825728 bytes 2024-04-06T08:31:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2712271872 bytes 2024-04-06T08:31:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2739951616 bytes 2024-04-06T08:31:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2790028288 bytes 2024-04-06T08:31:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2790644736 bytes 2024-04-06T08:31:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2909290496 bytes 2024-04-06T08:31:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2993420288 bytes 2024-04-06T08:31:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 3011013632 bytes 2024-04-06T08:31:38.514 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 3080612864 bytes 2024-04-06T08:31:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 3105820672 bytes 2024-04-06T08:31:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 3136404480 bytes 2024-04-06T08:31:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 3204164608 bytes 2024-04-06T08:31:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 3204953088 bytes 2024-04-06T08:31:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 3210813440 bytes 2024-04-06T08:31:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 3252233216 bytes 2024-04-06T08:31:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 3292434432 bytes 2024-04-06T08:31:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 3352481792 bytes 2024-04-06T08:31:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 3405512704 bytes 2024-04-06T08:31:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 3463102464 bytes 2024-04-06T08:31:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 3479929856 bytes 2024-04-06T08:31:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7900 IOs, 3535467520 bytes 2024-04-06T08:31:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8000 IOs, 3559672832 bytes 2024-04-06T08:31:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8100 IOs, 3662221312 bytes 2024-04-06T08:31:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8200 IOs, 3683175424 bytes 2024-04-06T08:31:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8300 IOs, 3725296640 bytes 2024-04-06T08:31:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8400 IOs, 3782917120 bytes 2024-04-06T08:31:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8500 IOs, 3791528960 bytes 2024-04-06T08:31:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8600 IOs, 3827423232 bytes 2024-04-06T08:31:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8700 IOs, 3838074880 bytes 2024-04-06T08:31:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8800 IOs, 3886812160 bytes 2024-04-06T08:31:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8900 IOs, 3911906304 bytes 2024-04-06T08:31:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9000 IOs, 3940245504 bytes 2024-04-06T08:31:41.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~87s 2024-04-06T08:31:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9100 IOs, 4017741824 bytes 2024-04-06T08:31:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9200 IOs, 4102838272 bytes 2024-04-06T08:31:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9300 IOs, 4147309568 bytes 2024-04-06T08:31:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9400 IOs, 4176361472 bytes 2024-04-06T08:31:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9500 IOs, 4185771008 bytes 2024-04-06T08:31:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9600 IOs, 4236224512 bytes 2024-04-06T08:31:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9700 IOs, 4242088960 bytes 2024-04-06T08:31:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9800 IOs, 4248210432 bytes 2024-04-06T08:31:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9900 IOs, 4376067072 bytes 2024-04-06T08:31:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10000 IOs, 4422970368 bytes 2024-04-06T08:31:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10100 IOs, 4477885440 bytes 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:31:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T08:31:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:31:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:31:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_9 2024-04-06T08:31:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T08:31:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:31:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b87b6af87c 2024-04-06T08:31:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:31:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_9 present 12b87b6af87c 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b87b6af87c 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:31:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:31:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T08:31:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:31:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:31:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_9 2024-04-06T08:31:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T08:31:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:31:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b87b6af87c 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b87b6af87c 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b87b6af87c -o 12b87b6af87c = 12b87b6af87c 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_9 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-04-06T08:31:43.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:31:43.250+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e94d9c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2024-04-06T08:31:43.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:31:43.250+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581e94d9c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=394, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2024-04-06T08:31:46.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2024-04-06T08:31:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:31:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-06T08:31:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:31:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:31:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:31:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:31:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:31:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T08:31:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:31:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:31:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_9 2024-04-06T08:31:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T08:31:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:31:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T08:31:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:31:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:31:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:31:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_9 2024-04-06T08:31:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:31:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:31:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_9 2024-04-06T08:31:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-06T08:31:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:31:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-06T08:31:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:31:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_9 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-04-06T08:31:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-06T08:31:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:31:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:31:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_9 2024-04-06T08:31:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:31:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:31:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_9 2024-04-06T08:31:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-06T08:31:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-06T08:31:52.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2024-04-06T08:31:57.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2024-04-06T08:32:01.118 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9725992564905509) > weight (0.25) 2024-04-06T08:32:01.118 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-06T08:32:01.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-06T08:32:01.119 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-06T08:32:01.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:32:01.122+0000 7fe2aa53b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 52891) UID: 1000 2024-04-06T08:32:03.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2024-04-06T08:32:03.120 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-06T08:32:03.120 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:32:03.121 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-06T08:32:03.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-06T08:32:03.121 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-06T08:32:03.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-06T08:32:03.124 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-04-06T08:32:08.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~114s 2024-04-06T08:32:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for i in `seq 1 10` 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image=image_10 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ create_image cluster2 mirror image_10 512M 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local size=128 2024-04-06T08:32:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -n 512M ']' 2024-04-06T08:32:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ size=512M 2024-04-06T08:32:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ shift 2024-04-06T08:32:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2024-04-06T08:32:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ enable_mirror cluster2 mirror image_10 2024-04-06T08:32:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:11.589 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local mode=snapshot 2024-04-06T08:32:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-04-06T08:32:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring enabled 2024-04-06T08:32:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-06T08:32:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:rbd image 'image_10': 2024-04-06T08:32:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: size 512 MiB in 128 objects 2024-04-06T08:32:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: order 22 (4 MiB objects) 2024-04-06T08:32:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: snapshot_count: 1 2024-04-06T08:32:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: id: 1a10a51272d3 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: block_name_prefix: rbd_data.1a10a51272d3 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: format: 2 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: features: layering, exclusive-lock 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: op_features: 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: flags: 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: create_timestamp: Sat Apr 6 08:32:11 2024 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: access_timestamp: Sat Apr 6 08:32:11 2024 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: modify_timestamp: Sat Apr 6 08:32:11 2024 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring state: enabled 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring mode: snapshot 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring global id: 8ece183b-72bc-4938-9ef9-e369ffabb7a0 2024-04-06T08:32:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout: mirroring primary: true 2024-04-06T08:32:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-06T08:32:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-06T08:32:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local duration=14 2024-04-06T08:32:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-06T08:32:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:100 IOs, 35600384 bytes 2024-04-06T08:32:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:200 IOs, 80878592 bytes 2024-04-06T08:32:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:300 IOs, 104457216 bytes 2024-04-06T08:32:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:400 IOs, 142710784 bytes 2024-04-06T08:32:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:500 IOs, 174501888 bytes 2024-04-06T08:32:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:600 IOs, 221690880 bytes 2024-04-06T08:32:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:700 IOs, 276773888 bytes 2024-04-06T08:32:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:800 IOs, 310066176 bytes 2024-04-06T08:32:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:900 IOs, 375386112 bytes 2024-04-06T08:32:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1000 IOs, 438131712 bytes 2024-04-06T08:32:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1100 IOs, 480939008 bytes 2024-04-06T08:32:13.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2024-04-06T08:32:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1200 IOs, 525144064 bytes 2024-04-06T08:32:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1300 IOs, 530134016 bytes 2024-04-06T08:32:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1400 IOs, 570852352 bytes 2024-04-06T08:32:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1500 IOs, 585493504 bytes 2024-04-06T08:32:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1600 IOs, 666477568 bytes 2024-04-06T08:32:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1700 IOs, 757355520 bytes 2024-04-06T08:32:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1800 IOs, 777058304 bytes 2024-04-06T08:32:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:1900 IOs, 810287104 bytes 2024-04-06T08:32:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2000 IOs, 836194304 bytes 2024-04-06T08:32:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2100 IOs, 903013376 bytes 2024-04-06T08:32:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2200 IOs, 937747456 bytes 2024-04-06T08:32:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2300 IOs, 961590272 bytes 2024-04-06T08:32:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2400 IOs, 1002763264 bytes 2024-04-06T08:32:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2500 IOs, 1089580032 bytes 2024-04-06T08:32:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2600 IOs, 1157785600 bytes 2024-04-06T08:32:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2700 IOs, 1193124864 bytes 2024-04-06T08:32:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2800 IOs, 1243324416 bytes 2024-04-06T08:32:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:2900 IOs, 1276210176 bytes 2024-04-06T08:32:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3000 IOs, 1276807168 bytes 2024-04-06T08:32:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3100 IOs, 1322704896 bytes 2024-04-06T08:32:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3200 IOs, 1348544512 bytes 2024-04-06T08:32:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3300 IOs, 1388851200 bytes 2024-04-06T08:32:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3400 IOs, 1434412032 bytes 2024-04-06T08:32:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3500 IOs, 1435046912 bytes 2024-04-06T08:32:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3600 IOs, 1472369664 bytes 2024-04-06T08:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3700 IOs, 1527896064 bytes 2024-04-06T08:32:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3800 IOs, 1604579328 bytes 2024-04-06T08:32:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:3900 IOs, 1651860480 bytes 2024-04-06T08:32:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4000 IOs, 1668765696 bytes 2024-04-06T08:32:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4100 IOs, 1709598720 bytes 2024-04-06T08:32:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4200 IOs, 1712346112 bytes 2024-04-06T08:32:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4300 IOs, 1757565952 bytes 2024-04-06T08:32:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4400 IOs, 1785266176 bytes 2024-04-06T08:32:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4500 IOs, 1797648384 bytes 2024-04-06T08:32:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4600 IOs, 1825968128 bytes 2024-04-06T08:32:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4700 IOs, 1908451328 bytes 2024-04-06T08:32:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4800 IOs, 2033048576 bytes 2024-04-06T08:32:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:4900 IOs, 2048821248 bytes 2024-04-06T08:32:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5000 IOs, 2097833984 bytes 2024-04-06T08:32:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5100 IOs, 2098459648 bytes 2024-04-06T08:32:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5200 IOs, 2137405440 bytes 2024-04-06T08:32:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5300 IOs, 2157101056 bytes 2024-04-06T08:32:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5400 IOs, 2185742336 bytes 2024-04-06T08:32:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5500 IOs, 2207611904 bytes 2024-04-06T08:32:19.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2024-04-06T08:32:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5600 IOs, 2252254208 bytes 2024-04-06T08:32:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5700 IOs, 2282636288 bytes 2024-04-06T08:32:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5800 IOs, 2306025472 bytes 2024-04-06T08:32:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:5900 IOs, 2341134336 bytes 2024-04-06T08:32:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6000 IOs, 2390341632 bytes 2024-04-06T08:32:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6100 IOs, 2403787776 bytes 2024-04-06T08:32:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6200 IOs, 2444014592 bytes 2024-04-06T08:32:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6300 IOs, 2446999552 bytes 2024-04-06T08:32:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6400 IOs, 2492862464 bytes 2024-04-06T08:32:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6500 IOs, 2533927936 bytes 2024-04-06T08:32:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6600 IOs, 2546083840 bytes 2024-04-06T08:32:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6700 IOs, 2588769280 bytes 2024-04-06T08:32:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6800 IOs, 2589386752 bytes 2024-04-06T08:32:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:6900 IOs, 2600805376 bytes 2024-04-06T08:32:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7000 IOs, 2747112448 bytes 2024-04-06T08:32:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7100 IOs, 2779254784 bytes 2024-04-06T08:32:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7200 IOs, 2804534272 bytes 2024-04-06T08:32:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7300 IOs, 2844644352 bytes 2024-04-06T08:32:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7400 IOs, 2882909184 bytes 2024-04-06T08:32:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7500 IOs, 2897571840 bytes 2024-04-06T08:32:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7600 IOs, 2969568256 bytes 2024-04-06T08:32:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7700 IOs, 3041673216 bytes 2024-04-06T08:32:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7800 IOs, 3084169216 bytes 2024-04-06T08:32:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:7900 IOs, 3146366976 bytes 2024-04-06T08:32:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8000 IOs, 3183036416 bytes 2024-04-06T08:32:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8100 IOs, 3225110528 bytes 2024-04-06T08:32:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8200 IOs, 3268075520 bytes 2024-04-06T08:32:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8300 IOs, 3287710720 bytes 2024-04-06T08:32:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8400 IOs, 3307182080 bytes 2024-04-06T08:32:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8500 IOs, 3388979200 bytes 2024-04-06T08:32:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8600 IOs, 3423787008 bytes 2024-04-06T08:32:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8700 IOs, 3481428992 bytes 2024-04-06T08:32:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8800 IOs, 3543901184 bytes 2024-04-06T08:32:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:8900 IOs, 3554793472 bytes 2024-04-06T08:32:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9000 IOs, 3598812160 bytes 2024-04-06T08:32:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9100 IOs, 3666108416 bytes 2024-04-06T08:32:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9200 IOs, 3715464192 bytes 2024-04-06T08:32:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9300 IOs, 3728574464 bytes 2024-04-06T08:32:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9400 IOs, 3777990656 bytes 2024-04-06T08:32:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9500 IOs, 3813984256 bytes 2024-04-06T08:32:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9600 IOs, 3845188608 bytes 2024-04-06T08:32:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9700 IOs, 3931858944 bytes 2024-04-06T08:32:24.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2024-04-06T08:32:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9800 IOs, 3989628928 bytes 2024-04-06T08:32:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:9900 IOs, 4017165312 bytes 2024-04-06T08:32:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10000 IOs, 4073271296 bytes 2024-04-06T08:32:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10100 IOs, 4164033536 bytes 2024-04-06T08:32:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10200 IOs, 4164799488 bytes 2024-04-06T08:32:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10300 IOs, 4234134528 bytes 2024-04-06T08:32:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10400 IOs, 4275059712 bytes 2024-04-06T08:32:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10500 IOs, 4321572864 bytes 2024-04-06T08:32:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10600 IOs, 4340574208 bytes 2024-04-06T08:32:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10700 IOs, 4368177152 bytes 2024-04-06T08:32:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:10800 IOs, 4517841920 bytes 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=124 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 124 -eq 124 ']' 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=present 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:32:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T08:32:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_10 2024-04-06T08:32:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T08:32:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:32:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id=12b8cdf06e10 2024-04-06T08:32:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:32:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:32:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_10 present 12b8cdf06e10 2024-04-06T08:32:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=present 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id=12b8cdf06e10 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_10 2024-04-06T08:32:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T08:32:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id=12b8cdf06e10 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 12b8cdf06e10 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -z 12b8cdf06e10 -o 12b8cdf06e10 = 12b8cdf06e10 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_state=present 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test present = present 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ disable_mirror cluster2 mirror image_10 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-04-06T08:32:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:27.773+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ed3b5000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2024-04-06T08:32:27.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:27.773+0000 7f744c074640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5581ed3b5000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=396, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=3aea22be-a2a5-4341-813f-81096758e7a9, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2024-04-06T08:32:28.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:32:28.293+0000 7f68b9cea640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fe099444e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-06T08:32:28.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:32:28.293+0000 7f68bfcf6640 -1 rbd::mirror::ImageReplayer: 0x55fe0a19ea00 [3/8ece183b-72bc-4938-9ef9-e369ffabb7a0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-06T08:32:30.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2024-04-06T08:32:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:Mirroring disabled 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local state=deleted 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local s 2024-04-06T08:32:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T08:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_10 2024-04-06T08:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T08:32:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:32:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T08:32:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ image_id= 2024-04-06T08:32:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-06T08:32:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0.1 2024-04-06T08:32:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-06T08:32:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local test_state=deleted 2024-04-06T08:32:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image_id= 2024-04-06T08:32:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_state=deleted 2024-04-06T08:32:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local current_image_id 2024-04-06T08:32:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ get_image_id cluster1 mirror image_10 2024-04-06T08:32:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local image=image_10 2024-04-06T08:32:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-06T08:32:32.950 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-06T08:32:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ current_image_id= 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n '' 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test deleted = deleted 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ purge_snapshots cluster2 mirror image_10 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-04-06T08:32:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-06T08:32:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-06T08:32:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ remove_image cluster2 mirror image_10 2024-04-06T08:32:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local image=image_10 2024-04-06T08:32:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-06T08:32:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-06T08:32:35.214 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9849366337784419) > weight (0.25) 2024-04-06T08:32:35.214 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-06T08:32:35.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-06T08:32:35.215 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-06T08:32:35.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:32:35.217+0000 7f68c0cf8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41479) UID: 1000 2024-04-06T08:32:35.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2024-04-06T08:32:40.215 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-06T08:32:40.216 INFO:teuthology.orchestra.run:waiting for 600 2024-04-06T08:32:40.216 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-06T08:32:40.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-06T08:32:40.217 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-06T08:32:40.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-06T08:32:40.220 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-04-06T08:32:40.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2024-04-06T08:32:46.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2024-04-06T08:32:51.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2024-04-06T08:32:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-06T08:32:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-04-06T08:32:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ tee -a /tmp/tmp.phadx0IwJQ/rbd-mirror.test.log 2024-04-06T08:32:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ date '+%F %T' 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo 2024-04-06 08:32:57 TEST: check if removed 'images'\''' OMAP are removed 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:2024-04-06 08:32:57 TEST: check if removed images' OMAP are removed 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local obj_name=rbd_mirroring 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local filter=status_global 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T08:32:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-04-06T08:32:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local filter=status_global 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-06T08:32:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -c status_global 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local obj_name=rbd_mirroring 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local filter=image_ 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local filter=image_ 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -c image_ 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-06T08:32:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:32:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local obj_name=rbd_mirror_leader 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local filter=image_map 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local filter=image_map 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster1 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -c image_map 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T08:32:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-04-06T08:32:57.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-04-06T08:32:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local obj_name=rbd_mirroring 2024-04-06T08:32:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local filter=status_global 2024-04-06T08:32:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T08:32:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster2 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local filter=status_global 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster2 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -c status_global 2024-04-06T08:32:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local obj_name=rbd_mirroring 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local filter=image_ 2024-04-06T08:32:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T08:32:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster2 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local filter=image_ 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster2 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -c image_ 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirroring 2024-04-06T08:32:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local pool=mirror 2024-04-06T08:32:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local obj_name=rbd_mirror_leader 2024-04-06T08:32:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local filter=image_map 2024-04-06T08:32:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-06T08:32:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ sleep 0 2024-04-06T08:32:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster2 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local filter=image_map 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local cluster=cluster2 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local pool=mirror 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ local obj_name=rbd_mirror_leader 2024-04-06T08:32:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ grep -c image_map 2024-04-06T08:32:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-04-06T08:32:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test 0 = 0 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ error_code=0 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set -e 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 0 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ cleanup 0 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local error_code=0 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ set +e 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -ne 0 ']' 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z '' ']' 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster instance 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T08:32:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-06T08:32:58.265 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror' does not exist 2024-04-06T08:32:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T08:32:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-06T08:32:58.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.861+0000 7f745988f640 -1 rbd::mirror::LeaderWatcher: 0x5581e9bd5b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-06T08:32:58.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f7453082640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5581ea41b200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:32:58.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f7453082640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5581ea41b680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:32:58.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f7453082640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5581ea41b8c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:32:58.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f7453082640 -1 rbd::mirror::PoolWatcher: 0x5581ea37e1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:32:58.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f7453082640 -1 rbd::mirror::PoolWatcher: 0x5581ea37e000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:32:58.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f7453082640 -1 rbd::mirror::PoolWatcher: 0x5581e9fabc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:32:58.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f744c074640 -1 rbd::mirror::PoolWatcher: 0x5581e9fabe00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:32:58.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f744c074640 -1 rbd::mirror::PoolWatcher: 0x5581e9faaf00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:32:58.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f744c074640 -1 rbd::mirror::PoolWatcher: 0x5581ea37e3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:32:58.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9bd5b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T08:32:58.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:32:58.865+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9bd5b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-06T08:32:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror' does not exist 2024-04-06T08:32:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T08:32:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-06T08:33:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror_parent' does not exist 2024-04-06T08:33:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ CEPH_ARGS= 2024-04-06T08:33:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-06T08:33:00.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdad2c65640 -1 rbd::mirror::LeaderWatcher: 0x557905417b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-06T08:33:00.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdacc458640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55790490efc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:33:00.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdacc458640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55790490ed80 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:33:00.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdacc458640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55790490f200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdacbc57640 -1 rbd::mirror::PoolWatcher: 0x557905461a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdacbc57640 -1 rbd::mirror::PoolWatcher: 0x5579054601e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdacbc57640 -1 rbd::mirror::PoolWatcher: 0x557905461e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdac5c4b640 -1 rbd::mirror::PoolWatcher: 0x5579051dda40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdac5c4b640 -1 rbd::mirror::PoolWatcher: 0x5579051dc000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdac5c4b640 -1 rbd::mirror::PoolWatcher: 0x557905461860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:00.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdad2c65640 -1 rbd::mirror::LeaderWatcher: 0x557905417b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T08:33:00.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:00.353+0000 7fdad2c65640 -1 rbd::mirror::LeaderWatcher: 0x557905417b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-06T08:33:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:pool 'mirror_parent' does not exist 2024-04-06T08:33:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-06T08:33:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirrors cluster1 2024-04-06T08:33:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1 2024-04-06T08:33:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 0 3 2024-04-06T08:33:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster1:0 '' 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:0 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster1:1 '' 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:1 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster1:2 '' 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:2 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster1:3 '' 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster1:3 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirrors cluster2 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:++ seq 0 3 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster2:0 '' 2024-04-06T08:33:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:0 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster2:1 '' 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:1 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster2:2 '' 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:2 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ stop_mirror cluster2:3 '' 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local cluster=cluster2:3 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ local sig= 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test -n 1 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ return 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' -z 1 ']' 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ test '' = /tmp/tmp.phadx0IwJQ 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ rm -Rf /tmp/tmp.phadx0IwJQ 2024-04-06T08:33:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ '[' 0 -eq 0 ']' 2024-04-06T08:33:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ echo OK 2024-04-06T08:33:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stdout:OK 2024-04-06T08:33:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi129.stderr:+ exit 0 2024-04-06T08:33:00.880 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-06T08:33:00.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-06T08:33:00.932 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-04-06T08:33:00.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-06T08:33:01.290 DEBUG:teuthology.parallel:result is None 2024-04-06T08:33:01.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-06T08:33:01.303 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-06T08:33:01.303 DEBUG:teuthology.orchestra.run.smithi129:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-06T08:33:01.362 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-06T08:33:01.362 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-06T08:33:01.373 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-06T08:33:01.373 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2024-04-06T08:33:01.373 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-06T08:33:01.373 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-06T08:33:01.399 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-06T08:33:01.399 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 19 2024-04-06T08:33:01.399 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-06T08:33:01.399 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:01.409 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-04-06T08:33:01.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:01.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.409+0000 7fdad3466640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 43673) UID: 1000 2024-04-06T08:33:01.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.409+0000 7fdad2464640 -1 rbd::mirror::LeaderWatcher: 0x557904ddc300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:01.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.409+0000 7fdad2c65640 -1 rbd::mirror::InstanceWatcher: 0x5579049ce380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5391 object: (2) No such file or directory 2024-04-06T08:33:01.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdad2464640 -1 rbd::mirror::InstanceWatcher: 0x5579049ce700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5391 object: (2) No such file or directory 2024-04-06T08:33:01.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdad2c65640 -1 rbd::mirror::InstanceWatcher: 0x557904e8e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5394 object: (2) No such file or directory 2024-04-06T08:33:01.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdabec3d640 -1 rbd::mirror::InstanceWatcher: 0x557904e8e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdacbc57640 -1 rbd::mirror::InstanceWatcher: 0x5579049ce380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdacbc57640 -1 rbd::mirror::InstanceWatcher: 0x5579049ce700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdab7c2f640 -1 rbd::mirror::MirrorStatusUpdater 0x557904c0ca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdad2464640 -1 rbd::mirror::NamespaceReplayer: 0x557904d3a680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdabec3d640 -1 rbd::mirror::MirrorStatusUpdater 0x557904bfc240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdad2c65640 -1 rbd::mirror::NamespaceReplayer: 0x557904d3a680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdac5c4b640 -1 rbd::mirror::MirrorStatusUpdater 0x557904a1e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdac5c4b640 -1 rbd::mirror::MirrorStatusUpdater 0x557904a1e480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdad2c65640 -1 rbd::mirror::NamespaceReplayer: 0x5579049c2820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.413+0000 7fdad2c65640 -1 rbd::mirror::NamespaceReplayer: 0x5579049c29c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.417+0000 7fdacc458640 -1 rbd::mirror::MirrorStatusUpdater 0x557904a1e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.417+0000 7fdacc458640 -1 rbd::mirror::MirrorStatusUpdater 0x557904a1e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.417+0000 7fdad2c65640 -1 rbd::mirror::NamespaceReplayer: 0x5579049c2820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.417+0000 7fdad2c65640 -1 rbd::mirror::NamespaceReplayer: 0x5579049c29c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdad2c65640 -1 rbd::mirror::LeaderWatcher: 0x557905417b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:01.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdad2c65640 -1 rbd::mirror::InstanceWatcher: 0x5579049ce000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5391 object: (2) No such file or directory 2024-04-06T08:33:01.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdacc458640 -1 rbd::mirror::InstanceWatcher: 0x5579049ce000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdac5c4b640 -1 rbd::mirror::MirrorStatusUpdater 0x55790492c480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdad2c65640 -1 rbd::mirror::NamespaceReplayer: 0x5579049c2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdacbc57640 -1 rbd::mirror::MirrorStatusUpdater 0x557905414a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi129.stderr:2024-04-06T08:33:01.421+0000 7fdad2464640 -1 rbd::mirror::NamespaceReplayer: 0x5579049c2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-04-06T08:33:01.439 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:01.450 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-04-06T08:33:01.450 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:01.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf78858640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 54462) UID: 1000 2024-04-06T08:33:01.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf78057640 -1 rbd::mirror::InstanceWatcher: 0x55e243b82380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5947 object: (2) No such file or directory 2024-04-06T08:33:01.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf78057640 -1 rbd::mirror::InstanceWatcher: 0x55e243b82700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5947 object: (2) No such file or directory 2024-04-06T08:33:01.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf7184a640 -1 rbd::mirror::InstanceWatcher: 0x55e243b82380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf7184a640 -1 rbd::mirror::InstanceWatcher: 0x55e243b82700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf6b03d640 -1 rbd::mirror::MirrorStatusUpdater 0x55e243bd4360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf78057640 -1 rbd::mirror::NamespaceReplayer: 0x55e243b6c820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf6a83c640 -1 rbd::mirror::MirrorStatusUpdater 0x55e243bd4480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf77856640 -1 rbd::mirror::NamespaceReplayer: 0x55e243b6c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf71049640 -1 rbd::mirror::MirrorStatusUpdater 0x55e243bd4000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf78057640 -1 rbd::mirror::NamespaceReplayer: 0x55e243b6c820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf7184a640 -1 rbd::mirror::MirrorStatusUpdater 0x55e243bd4120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.485+0000 7fbf77856640 -1 rbd::mirror::NamespaceReplayer: 0x55e243b6c9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:01.833+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9370300 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-06T08:33:01.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:01.833+0000 7f7445066640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5581e940e6c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:33:01.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:01.833+0000 7f743f05a640 -1 rbd::mirror::PoolWatcher: 0x5581e9faa3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:01.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:01.837+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9370300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T08:33:01.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:01.837+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9370300 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-06T08:33:01.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.849+0000 7fbf6402f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e243ae2fc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-06T08:33:01.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.849+0000 7fbf5d822640 -1 rbd::mirror::PoolWatcher: 0x55e2441d92c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-06T08:33:01.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.849+0000 7fbf77856640 -1 rbd::mirror::LeaderWatcher: 0x55e243f82300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-06T08:33:01.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.849+0000 7fbf77856640 -1 rbd::mirror::LeaderWatcher: 0x55e243f82300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-04-06T08:33:01.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.849+0000 7fbf78057640 -1 rbd::mirror::LeaderWatcher: 0x55e243f82300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:01.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.849+0000 7fbf78057640 -1 rbd::mirror::InstanceWatcher: 0x55e24403a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5949 object: (2) No such file or directory 2024-04-06T08:33:01.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.853+0000 7fbf6382e640 -1 rbd::mirror::InstanceWatcher: 0x55e24403a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.853+0000 7fbf5d021640 -1 rbd::mirror::MirrorStatusUpdater 0x55e243dbca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.853+0000 7fbf78057640 -1 rbd::mirror::NamespaceReplayer: 0x55e243f884e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.853+0000 7fbf6402f640 -1 rbd::mirror::MirrorStatusUpdater 0x55e243db8240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.853+0000 7fbf77856640 -1 rbd::mirror::NamespaceReplayer: 0x55e243f884e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.857+0000 7fbf78057640 -1 rbd::mirror::LeaderWatcher: 0x55e24393db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:01.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.857+0000 7fbf78057640 -1 rbd::mirror::InstanceWatcher: 0x55e243b82000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5947 object: (2) No such file or directory 2024-04-06T08:33:01.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.861+0000 7fbf71049640 -1 rbd::mirror::InstanceWatcher: 0x55e243b82000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.861+0000 7fbf6b03d640 -1 rbd::mirror::MirrorStatusUpdater 0x55e2445b86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.861+0000 7fbf78057640 -1 rbd::mirror::NamespaceReplayer: 0x55e243b6c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.861+0000 7fbf7184a640 -1 rbd::mirror::MirrorStatusUpdater 0x55e24455e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi129.stderr:2024-04-06T08:33:01.861+0000 7fbf77856640 -1 rbd::mirror::NamespaceReplayer: 0x55e243b6c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:02.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.485+0000 7fee18bd7640 -1 rbd::mirror::RemotePoolPoller: 0x564dca48ad00 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-06T08:33:02.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.485+0000 7fee183d6640 -1 rbd::mirror::RemotePoolPoller: 0x564dca48ad00 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-06T08:33:02.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.485+0000 7fee183d6640 -1 rbd::mirror::PoolReplayer: 0x564dcb34a000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-06T08:33:02.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee253f0640 -1 rbd::mirror::InstanceWatcher: 0x564dcb3a8380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4858 object: (2) No such file or directory 2024-04-06T08:33:02.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee253f0640 -1 rbd::mirror::InstanceWatcher: 0x564dcb3a8700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4858 object: (2) No such file or directory 2024-04-06T08:33:02.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee1ebe3640 -1 rbd::mirror::InstanceWatcher: 0x564dcb3a8380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:02.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee1ebe3640 -1 rbd::mirror::InstanceWatcher: 0x564dcb3a8700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:02.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee183d6640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcb420480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:02.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee183d6640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcb4205a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:02.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee253f0640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb3a4820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:02.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee253f0640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb3a49c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:02.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee1ebe3640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcb420000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:02.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee1ebe3640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcb420120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:02.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee25bf1640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb3a4820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:02.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.489+0000 7fee25bf1640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb3a49c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:02.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2024-04-06T08:33:02.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.681+0000 7fee0abbb640 -1 rbd::mirror::RemotePoolPoller: 0x564dca48b860 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-06T08:33:02.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.681+0000 7fee0b3bc640 -1 rbd::mirror::RemotePoolPoller: 0x564dca48b860 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-06T08:33:02.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:02.681+0000 7fee0b3bc640 -1 rbd::mirror::PoolReplayer: 0x564dcb3b6000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-06T08:33:03.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.169+0000 7fee253f0640 -1 rbd::mirror::LeaderWatcher: 0x564dcc02db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:03.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.177+0000 7fee253f0640 -1 rbd::mirror::InstanceWatcher: 0x564dcb3a8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4858 object: (2) No such file or directory 2024-04-06T08:33:03.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.177+0000 7fee1f3e4640 -1 rbd::mirror::InstanceWatcher: 0x564dcb3a8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:03.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.177+0000 7fee183d6640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcc0145a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.177+0000 7fee253f0640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb3a4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.177+0000 7fee1ebe3640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcbf9a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.177+0000 7fee25bf1640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb3a4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.185+0000 7fee253f0640 -1 rbd::mirror::LeaderWatcher: 0x564dcb7b6300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:03.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.189+0000 7fee253f0640 -1 rbd::mirror::InstanceWatcher: 0x564dcb86a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4864 object: (2) No such file or directory 2024-04-06T08:33:03.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.189+0000 7fee11bc9640 -1 rbd::mirror::InstanceWatcher: 0x564dcb86a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:03.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.189+0000 7fee0b3bc640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcb6047e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.189+0000 7fee253f0640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb7bc4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.189+0000 7fee113c8640 -1 rbd::mirror::MirrorStatusUpdater 0x564dcb5be120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:03.189+0000 7fee25bf1640 -1 rbd::mirror::NamespaceReplayer: 0x564dcb7bc4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.273+0000 7f555dbc0640 -1 rbd::mirror::RemotePoolPoller: 0x55777fffef70 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-06T08:33:03.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.277+0000 7f555e3c1640 -1 rbd::mirror::RemotePoolPoller: 0x55777fffef70 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-06T08:33:03.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.277+0000 7f555e3c1640 -1 rbd::mirror::PoolReplayer: 0x557780ec8000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-06T08:33:03.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.277+0000 7f556abda640 -1 rbd::mirror::InstanceWatcher: 0x557781148380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6179 object: (2) No such file or directory 2024-04-06T08:33:03.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.277+0000 7f556abda640 -1 rbd::mirror::InstanceWatcher: 0x557781148700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6179 object: (2) No such file or directory 2024-04-06T08:33:03.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.277+0000 7f55643cd640 -1 rbd::mirror::InstanceWatcher: 0x557781148380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:03.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.277+0000 7f55643cd640 -1 rbd::mirror::InstanceWatcher: 0x557781148700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:03.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f555dbc0640 -1 rbd::mirror::MirrorStatusUpdater 0x5577811b05a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f555dbc0640 -1 rbd::mirror::MirrorStatusUpdater 0x5577811b0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f556abda640 -1 rbd::mirror::NamespaceReplayer: 0x557781146820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f556abda640 -1 rbd::mirror::NamespaceReplayer: 0x5577811469c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f5564bce640 -1 rbd::mirror::MirrorStatusUpdater 0x5577811b0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f5564bce640 -1 rbd::mirror::MirrorStatusUpdater 0x5577811b0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f556b3db640 -1 rbd::mirror::NamespaceReplayer: 0x557781146820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.281+0000 7f556b3db640 -1 rbd::mirror::NamespaceReplayer: 0x5577811469c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.593+0000 7f5550ba6640 -1 rbd::mirror::RemotePoolPoller: 0x55777ffff6c0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-06T08:33:03.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.597+0000 7f55503a5640 -1 rbd::mirror::RemotePoolPoller: 0x55777ffff6c0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-06T08:33:03.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.597+0000 7f55503a5640 -1 rbd::mirror::PoolReplayer: 0x557781152000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-06T08:33:03.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.645+0000 7f556b3db640 -1 rbd::mirror::LeaderWatcher: 0x557781badb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:03.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.645+0000 7f556b3db640 -1 rbd::mirror::InstanceWatcher: 0x557781148000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6179 object: (2) No such file or directory 2024-04-06T08:33:03.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.645+0000 7f5564bce640 -1 rbd::mirror::InstanceWatcher: 0x557781148000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:03.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.649+0000 7f555e3c1640 -1 rbd::mirror::MirrorStatusUpdater 0x557781b946c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.649+0000 7f556b3db640 -1 rbd::mirror::NamespaceReplayer: 0x557781146000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.649+0000 7f5564bce640 -1 rbd::mirror::MirrorStatusUpdater 0x557781b1a7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.649+0000 7f556abda640 -1 rbd::mirror::NamespaceReplayer: 0x557781146000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.653+0000 7f556b3db640 -1 rbd::mirror::LeaderWatcher: 0x55778152a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:03.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.653+0000 7f556b3db640 -1 rbd::mirror::InstanceWatcher: 0x557781148a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6182 object: (2) No such file or directory 2024-04-06T08:33:03.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.657+0000 7f5556bb2640 -1 rbd::mirror::InstanceWatcher: 0x557781148a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:03.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.657+0000 7f5550ba6640 -1 rbd::mirror::MirrorStatusUpdater 0x5577813747e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.657+0000 7f556b3db640 -1 rbd::mirror::NamespaceReplayer: 0x557781492680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.657+0000 7f5556bb2640 -1 rbd::mirror::MirrorStatusUpdater 0x55778136a240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:03.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:03.657+0000 7f556abda640 -1 rbd::mirror::NamespaceReplayer: 0x557781492680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:03.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:03.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:03.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:03.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:04.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:04.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:04.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.679 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.680 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.707 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.711 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.749 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.797 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.818 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.825+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.825+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.924 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.961 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.971 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:05.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:05.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:05.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.037+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.167 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.200 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.273+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.277+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.281+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.285+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.289+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.293+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.297+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.301+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.305+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.308 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.309+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.313+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.317+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.321+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.325+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.329+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.333+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.337+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.341+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.345+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.349+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.353+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.357+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.361+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.365+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.369+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.373+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.377+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.381+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.385+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.389+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.393+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.397+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.401+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.405+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.409+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.413+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.417+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.421+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.425+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.429+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.433+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.437+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.441+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.445+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.449+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.453+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.457+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.461+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.465+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.469+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.473+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.477+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.481+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.485+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.489+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.493+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.497+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.501+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.505+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.509+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.513+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.517+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.521+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.525+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.529+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.533+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.537+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.541+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.545+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.549+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.553+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.557+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.561+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.565+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.569+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.573+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.577+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.581+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.585+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.589+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.593+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.597+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.601+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.606 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.605+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.607 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.609+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.613 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.613+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.615 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.617+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.621+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.625+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.629+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.633+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.637+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.641+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.642 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.641+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.643 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.645+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.649+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.653+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.657+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.662 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.661+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.665 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.665+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.669+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.673+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.677 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.681 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.677+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.681+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.685 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.687 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.685+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.689+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.693 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.695 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.693+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.698 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.697+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.700 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.701+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.705 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.706 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.705+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.709+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.713+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.717+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.721+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.725+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.729+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.733+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.737+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.741+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.747 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.745+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.748 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.749+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.753+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.757+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.760 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.762 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.761+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.764 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.765+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.768 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.769+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.773+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.777+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.781+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.785+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.789+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.793 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.794 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.795 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.793+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.799 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.797+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.801+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.807 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.805+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.809+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.812 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.813 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.813+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.817+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.819 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.821+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.827 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.825+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.829+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.833+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.839 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.837+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.841+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.845 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.846 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.845+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.849+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.853+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.857 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.858 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.857+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.861+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.865+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.869+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.873+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.877+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.881+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.885+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.889+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.893+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.897+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.901+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.905+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.909+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.914 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.915 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.913+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.917+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.921 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.921+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.923 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.925 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.926 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.925+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.928 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.929+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.933+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.937+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.941+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.945+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.950 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.949+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.953+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.957+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.961+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.965+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.969+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.973 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.973+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.977 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.977+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.981+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.985+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.989+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.993+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:06.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:06.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:06.997+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.001+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.005+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.009+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.013+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.017+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.021+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.025+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.029+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.033+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.037+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.042 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.041+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.045+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.049+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.053+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.057+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.061+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.065+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.069+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.073+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.077+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.082 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.081+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.085+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.089+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.093 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.093+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.095 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.096 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.097+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.101+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.105+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.109+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.113 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.113+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.117+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.121+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.125+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.130 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.129+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.133+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.137+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.141+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.145+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.149+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.153+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.157+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.161+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.165 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.165+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.169+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.173 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.173+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.177+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.181+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.185+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.189+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.193+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.197+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.199 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.201+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.205+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.205+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.209+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.213+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.217+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.221+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.227 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.225+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.229+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.233+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.237+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.241+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.245+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.249+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.252 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.253+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.257+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.261+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.261+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.266 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.265+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.269+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.272+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.276+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.280+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.284+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.290 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.288+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.291 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.292+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.300 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.296+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.300+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.304+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.308+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.308+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.308+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.308+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.308+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.308+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.312+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.316+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.322 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.320+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.324+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.328+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.333 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.332+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.336+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.340+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.345 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.344+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.348+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.352+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.356+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.360+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.364+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.368+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.372+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.376+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.380+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.384+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.388+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.392+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.396+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.400+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.404+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.408+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.408+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.408+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.408+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.408+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.408+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.412+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.416+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.420+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.424+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.424+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.424+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.424+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.424+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.424+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.430 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.428+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.432+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.436+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.440+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.447 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.444+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.448+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.452+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.456+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.460+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.464+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.468+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.472+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.476+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.480+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.484+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.488+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.492+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.496+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.500+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.504+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.508+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.512+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.516+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.516+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.516+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.516+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.516+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.516+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.520+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.525 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.524+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.528+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.532+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.536+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.540+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.546 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.544+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.547 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.549 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-04-06T08:33:07.551 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:07.563 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-04-06T08:33:07.563 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:07.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.548+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.552+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.556+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.560+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5ce2d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 64820) UID: 1000 2024-04-06T08:33:07.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5be2b640 -1 rbd::mirror::LeaderWatcher: 0x55fa1b044300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:07.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5561e640 -1 rbd::mirror::InstanceWatcher: 0x55fa1ac44700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5561e640 -1 rbd::mirror::InstanceWatcher: 0x55fa1ac44380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5c62c640 -1 rbd::mirror::InstanceWatcher: 0x55fa1ac44a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6703 object: (2) No such file or directory 2024-04-06T08:33:07.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.564+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf47e03640 -1 rbd::mirror::InstanceWatcher: 0x55fa1ac44a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf4f612640 -1 rbd::mirror::MirrorStatusUpdater 0x55fa1ac8e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf4f612640 -1 rbd::mirror::MirrorStatusUpdater 0x55fa1ac8e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5be2b640 -1 rbd::mirror::NamespaceReplayer: 0x55fa1ac3a820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.564+0000 7faf5be2b640 -1 rbd::mirror::NamespaceReplayer: 0x55fa1ac3a9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.568+0000 7faf47e03640 -1 rbd::mirror::MirrorStatusUpdater 0x55fa1aea86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.568+0000 7faf5c62c640 -1 rbd::mirror::NamespaceReplayer: 0x55fa1afa6680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.568+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.572+0000 7faf5561e640 -1 rbd::mirror::InstanceWatcher: 0x55fa1ac44000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.572+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.572+0000 7faf4ee11640 -1 rbd::mirror::MirrorStatusUpdater 0x55fa1b68a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.572+0000 7faf5be2b640 -1 rbd::mirror::NamespaceReplayer: 0x55fa1ac3a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.576+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.580+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.584+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.588+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.594 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.592+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.596+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-04-06T08:33:07.600 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:07.617 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-04-06T08:33:07.617 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:07.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.600+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.621 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.604+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.623 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.608+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.612+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86410f5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 63885) UID: 1000 2024-04-06T08:33:07.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86408f4640 -1 rbd::mirror::LeaderWatcher: 0x564eb9a4a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:07.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.616+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f863a0e7640 -1 rbd::mirror::InstanceWatcher: 0x564eb963c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f863a0e7640 -1 rbd::mirror::InstanceWatcher: 0x564eb963c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f862c8cc640 -1 rbd::mirror::InstanceWatcher: 0x564eb963ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86330d9640 -1 rbd::mirror::MirrorStatusUpdater 0x564eb968a7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86330d9640 -1 rbd::mirror::MirrorStatusUpdater 0x564eb968a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86408f4640 -1 rbd::mirror::NamespaceReplayer: 0x564eb962e820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86408f4640 -1 rbd::mirror::NamespaceReplayer: 0x564eb962e9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f862c0cb640 -1 rbd::mirror::MirrorStatusUpdater 0x564eb98a2240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86400f3640 -1 rbd::mirror::NamespaceReplayer: 0x564eb9a504e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86408f4640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86408f4640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f863a0e7640 -1 rbd::mirror::MirrorStatusUpdater 0x564eb968a000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86400f3640 -1 rbd::mirror::NamespaceReplayer: 0x564eb962e820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.620+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f86400f3640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f86408f4640 -1 rbd::mirror::LeaderWatcher: 0x564eba2a7b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:07.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f86408f4640 -1 rbd::mirror::InstanceWatcher: 0x564eb963c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6615 object: (2) No such file or directory 2024-04-06T08:33:07.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.624+0000 7f86398e6640 -1 rbd::mirror::InstanceWatcher: 0x564eb963c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f86330d9640 -1 rbd::mirror::MirrorStatusUpdater 0x564eba2906c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f86408f4640 -1 rbd::mirror::NamespaceReplayer: 0x564eb962e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f863a0e7640 -1 rbd::mirror::MirrorStatusUpdater 0x564eba2326c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.638 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f86400f3640 -1 rbd::mirror::NamespaceReplayer: 0x564eb962e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.628+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.632+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.632+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.632+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.632+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.632+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.632+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.636+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.640+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.640+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.640+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.640+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.640+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.640+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.644+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-04-06T08:33:07.651 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:07.662 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-04-06T08:33:07.662 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:07.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.648+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.652+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.652+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.652+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.652+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.652+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.652+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.656+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.660+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.660+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.660+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.660+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.660+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.660+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi129.stderr:2024-04-06T08:33:07.660+0000 7fee263f2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 42783) UID: 1000 2024-04-06T08:33:07.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.664+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.664+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.664+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.664+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.664+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.664+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.668+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.672+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.676+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-04-06T08:33:07.683 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:07.694 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-04-06T08:33:07.694 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:07.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.680+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.684+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.688+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.692+0000 7f242a131640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 63623) UID: 1000 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f242912f640 -1 rbd::mirror::InstanceWatcher: 0x562ac9488380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6127 object: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2429930640 -1 rbd::mirror::InstanceWatcher: 0x562ac9488000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6127 object: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2422922640 -1 rbd::mirror::InstanceWatcher: 0x562ac9488380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2415107640 -1 rbd::mirror::InstanceWatcher: 0x562ac968a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2423123640 -1 rbd::mirror::InstanceWatcher: 0x562ac9488000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2423123640 -1 rbd::mirror::MirrorStatusUpdater 0x562ac920c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2429930640 -1 rbd::mirror::NamespaceReplayer: 0x562ac918e820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2422922640 -1 rbd::mirror::MirrorStatusUpdater 0x562ac920c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f242912f640 -1 rbd::mirror::NamespaceReplayer: 0x562ac918e9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2429930640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.696+0000 7f2429930640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f2429930640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f2429930640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2024-04-06T08:33:07.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f2429930640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2024-04-06T08:33:07.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f242912f640 -1 rbd::mirror::LeaderWatcher: 0x562ac9dedb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:07.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f242912f640 -1 rbd::mirror::InstanceWatcher: 0x562ac9e65180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6127 object: (2) No such file or directory 2024-04-06T08:33:07.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f2423123640 -1 rbd::mirror::InstanceWatcher: 0x562ac9e65180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f241c115640 -1 rbd::mirror::MirrorStatusUpdater 0x562ac9dd86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.700+0000 7f242912f640 -1 rbd::mirror::NamespaceReplayer: 0x562ac918e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.704+0000 7f2423123640 -1 rbd::mirror::MirrorStatusUpdater 0x562ac9d5c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi129.stderr:2024-04-06T08:33:07.704+0000 7f2429930640 -1 rbd::mirror::NamespaceReplayer: 0x562ac918e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-04-06T08:33:07.726 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:07.737 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-04-06T08:33:07.737 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:07.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi129.stderr:2024-04-06T08:33:07.768+0000 7f556bbdc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 64478) UID: 1000 2024-04-06T08:33:07.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-04-06T08:33:07.785 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-06T08:33:07.797 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-04-06T08:33:07.797 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:33:07.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f745a090640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41142) UID: 1000 2024-04-06T08:33:07.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9370300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:07.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f745908e640 -1 rbd::mirror::InstanceWatcher: 0x5581e8f68380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4792 object: (2) No such file or directory 2024-04-06T08:33:07.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f745908e640 -1 rbd::mirror::InstanceWatcher: 0x5581e8f68a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4792 object: (2) No such file or directory 2024-04-06T08:33:07.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f745988f640 -1 rbd::mirror::InstanceWatcher: 0x5581e942c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4794 object: (2) No such file or directory 2024-04-06T08:33:07.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f7452881640 -1 rbd::mirror::InstanceWatcher: 0x5581e8f68380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f7452881640 -1 rbd::mirror::InstanceWatcher: 0x5581e8f68a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.832+0000 7f7445867640 -1 rbd::mirror::InstanceWatcher: 0x5581e942c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f744c074640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e8fc8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f744c074640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e8fc8480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f745908e640 -1 rbd::mirror::NamespaceReplayer: 0x5581e8f66820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f743f05a640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e8fc86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f745908e640 -1 rbd::mirror::NamespaceReplayer: 0x5581e8f669c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f745908e640 -1 rbd::mirror::NamespaceReplayer: 0x5581e92d6680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f7452881640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e8fc8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f7452881640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e8fc8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f745908e640 -1 rbd::mirror::NamespaceReplayer: 0x5581e8f669c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.836+0000 7f745908e640 -1 rbd::mirror::NamespaceReplayer: 0x5581e8f66820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.840+0000 7f745908e640 -1 rbd::mirror::LeaderWatcher: 0x5581e9bd5b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-06T08:33:07.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.840+0000 7f745908e640 -1 rbd::mirror::InstanceWatcher: 0x5581e8f68000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4792 object: (2) No such file or directory 2024-04-06T08:33:07.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.840+0000 7f7453082640 -1 rbd::mirror::InstanceWatcher: 0x5581e8f68000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-06T08:33:07.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.840+0000 7f744c074640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e9bbc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.840+0000 7f745908e640 -1 rbd::mirror::NamespaceReplayer: 0x5581e8f66000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.844+0000 7f7453082640 -1 rbd::mirror::MirrorStatusUpdater 0x5581e9b4e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-06T08:33:07.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi129.stderr:2024-04-06T08:33:07.844+0000 7f745988f640 -1 rbd::mirror::NamespaceReplayer: 0x5581e8f66000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-06T08:33:07.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-04-06T08:33:07.870 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-06T08:33:07.881 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-06T08:33:07.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:07.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2024-04-06T08:33:08.213 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:08.214 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:08.231 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-04-06T08:33:07.207404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":198,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2266,"num_read_kb":-2150,"num_write":-427,"num_write_kb":-329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.451505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.272368+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.272368+0000","last_peered":"2024-04-06T08:33:00.272368+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.272368+0000","last_undegraded":"2024-04-06T08:33:00.272368+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":1.74}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:08.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:08.522 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:08.523 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:08.542 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-04-06T08:33:07.207404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":198,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2266,"num_read_kb":-2150,"num_write":-427,"num_write_kb":-329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.451505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.272368+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.272368+0000","last_peered":"2024-04-06T08:33:00.272368+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.272368+0000","last_undegraded":"2024-04-06T08:33:00.272368+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":1.74}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:08.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:08.821 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:08.838 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-04-06T08:33:07.207404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":198,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":78,"apply_latency_ms":78,"commit_latency_ns":78000000,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2266,"num_read_kb":-2150,"num_write":-427,"num_write_kb":-329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.451505"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":464,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.272368+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.272368+0000","last_peered":"2024-04-06T08:33:00.272368+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.272368+0000","last_undegraded":"2024-04-06T08:33:00.272368+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":1.74}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:08.839 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:08.839 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 464, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.272368+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.272368+0000', 'last_peered': '2024-04-06T08:33:00.272368+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.272368+0000', 'last_undegraded': '2024-04-06T08:33:00.272368+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:08.839 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:08.839 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:08.840 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:08.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:09.156 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:09.156 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2360,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":47,"modified":"2024-04-06T08:33:03.206075+0000","services":{"rbd-mirror":{"daemons":{"summary":"","6612":{"start_epoch":44,"start_stamp":"2024-04-06T08:30:40.311344+0000","gid":6612,"addr":"172.21.15.129:0/2939384077","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable)","ceph_version_short":"18.2.2-1059-g4d05f4ca","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi129","id":"mirror.0","instance_id":"6612","kernel_description":"#111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024","kernel_version":"5.15.0-101-generic","mem_swap_kb":"2097148","mem_total_kb":"32746920","os":"Linux"},"task_status":{}},"6697":{"start_epoch":46,"start_stamp":"2024-04-06T08:32:40.315452+0000","gid":6697,"addr":"172.21.15.129:0/1370028485","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.2.2-1059-g4d05f4ca (4d05f4cad464de0fcdd11a2856643e533aea34e8) reef (stable)","ceph_version_short":"18.2.2-1059-g4d05f4ca","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi129","id":"mirror.1","instance_id":"6697","kernel_description":"#111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024","kernel_version":"5.15.0-101-generic","mem_swap_kb":"2097148","mem_total_kb":"32746920","os":"Linux"},"task_status":{}}}}}},"progress_events":{}} 2024-04-06T08:33:09.174 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:09.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:09.438 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:09.439 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:09.455 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1623,"stamp":"2024-04-06T08:33:09.208168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2268,"num_read_kb":-2152,"num_write":-427,"num_write_kb":-329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004189"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.69499999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.5},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:12.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:12.746 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:12.747 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:12.763 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-04-06T08:33:11.208804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.448027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.69499999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.5},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:12.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:13.041 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:13.042 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-04-06T08:33:11.208804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.448027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.69499999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.5},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:13.060 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:13.061 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:13.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:13.345 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2024-04-06T08:33:13.389 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:13.389 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2364,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:13.407 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:13.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:13.693 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:13.694 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:13.712 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1625,"stamp":"2024-04-06T08:33:13.209396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004126"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.69499999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.5},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:16.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:17.003 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:17.020 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-04-06T08:33:15.209986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003954"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.47399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:17.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:17.298 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:17.298 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:17.315 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2024-04-06T08:33:17.210618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003852"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.47399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:17.316 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:17.317 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:17.317 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:17.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:17.645 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:17.645 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2368,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:17.663 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:17.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:17.929 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:17.930 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:17.947 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2024-04-06T08:33:17.210618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003852"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.47399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:18.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2024-04-06T08:33:20.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:21.246 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2024-04-06T08:33:21.211834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:21.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:21.553 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:21.553 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:21.571 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2024-04-06T08:33:21.211834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:21.571 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:21.572 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:21.573 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:21.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:21.910 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2373,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:21.927 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:21.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:22.212 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:22.231 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2024-04-06T08:33:21.211834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.54200000000000004,"15min":0.56499999999999995},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":5.8970000000000002,"5min":5.8970000000000002,"15min":11.275},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.61699999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.9340000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:24.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2024-04-06T08:33:25.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:25.532 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:25.532 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:25.548 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1631,"stamp":"2024-04-06T08:33:25.213022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:25.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:25.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:25.835 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:25.853 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1631,"stamp":"2024-04-06T08:33:25.213022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:25.854 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:25.855 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:25.855 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:25.855 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:25.855 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:25.855 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:25.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:26.201 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:26.201 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2377,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:26.220 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:26.221 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:26.509 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:26.527 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1631,"stamp":"2024-04-06T08:33:25.213022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":2.9009999999999998,"5min":12.539999999999999,"15min":12.539999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.95099999999999996,"15min":0.74299999999999999},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":2.1400000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:29.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:29.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2024-04-06T08:33:29.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:29.826 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:29.842 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-04-06T08:33:29.214264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003645"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:29.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:30.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:30.120 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:30.136 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-04-06T08:33:29.214264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003645"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:30.137 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:30.138 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:30.138 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:30.138 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:30.138 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:30.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:30.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:30.467 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2381,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:30.484 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:30.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:30.750 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:30.751 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:30.766 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-04-06T08:33:29.214264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003645"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:33.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:34.054 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:34.055 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:34.071 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-04-06T08:33:33.215481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003647"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:34.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:34.342 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:34.342 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:34.359 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-04-06T08:33:33.215481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003647"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:34.360 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:34.360 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:34.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:34.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:34.680 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:34.681 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2385,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:34.698 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:34.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:34.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2024-04-06T08:33:34.972 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:34.972 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:34.990 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-04-06T08:33:33.215481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003647"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:37.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:38.283 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:38.300 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-04-06T08:33:37.216690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.61299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.502}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:38.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:38.576 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:38.576 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:38.592 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-04-06T08:33:37.216690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.61299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.502}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:38.593 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:38.593 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:38.594 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:38.595 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:38.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:38.909 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:38.909 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2390,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:38.927 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:38.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:39.193 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:39.194 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:39.211 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-04-06T08:33:37.216690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003667"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.773867+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:32:58.773867+0000","last_peered":"2024-04-06T08:32:58.773867+0000","last_clean":"2024-04-06T08:32:58.773867+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:32:58.773867+0000","last_undegraded":"2024-04-06T08:32:58.773867+0000","last_fullsized":"2024-04-06T08:32:58.773867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.61299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.502}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:40.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2024-04-06T08:33:42.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:42.516 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:42.516 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:42.534 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-04-06T08:33:41.217890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.28699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:42.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:42.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:42.827 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-04-06T08:33:41.217890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.28699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:42.845 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.261555+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:00.261555+0000', 'last_peered': '2024-04-06T08:33:00.261555+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:00.261555+0000', 'last_undegraded': '2024-04-06T08:33:00.261555+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:42.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:42.847 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:42.847 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:42.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:43.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:43.191 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2394,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:43.210 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:43.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:43.476 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:43.477 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:43.493 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1640,"stamp":"2024-04-06T08:33:43.218484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.261555+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:00.261555+0000","last_peered":"2024-04-06T08:33:00.261555+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:00.261555+0000","last_undegraded":"2024-04-06T08:33:00.261555+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":460,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.267771+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.267771+0000","last_peered":"2024-04-06T08:33:00.267771+0000","last_clean":"2024-04-06T08:33:00.267771+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.267771+0000","last_undegraded":"2024-04-06T08:33:00.267771+0000","last_fullsized":"2024-04-06T08:33:00.267771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.63900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.28699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:45.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2024-04-06T08:33:46.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:46.788 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:46.789 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:46.805 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-04-06T08:33:45.219056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.63},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.72399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.443},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.47699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:46.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:47.074 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:47.075 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:47.091 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-04-06T08:33:45.219056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.63},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.72399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.443},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.47699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.273863+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.273863+0000', 'last_peered': '2024-04-06T08:33:00.273863+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.273863+0000', 'last_undegraded': '2024-04-06T08:33:00.273863+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:47.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:47.093 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:47.093 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:47.093 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:47.093 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:47.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:47.422 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:47.422 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2398,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:47.439 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:47.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:47.717 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:47.717 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:47.734 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1642,"stamp":"2024-04-06T08:33:47.219724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.273863+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.273863+0000","last_peered":"2024-04-06T08:33:00.273863+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.273863+0000","last_undegraded":"2024-04-06T08:33:00.273863+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.63},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.72399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.443},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.47699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:50.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:51.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:51.038 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:51.056 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-04-06T08:33:49.220337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003647"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.66200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:51.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:51.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2024-04-06T08:33:51.350 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:51.351 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:51.369 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1644,"stamp":"2024-04-06T08:33:51.220969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.66200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:51.370 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:51.370 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:51.370 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:51.370 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:51.370 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:51.370 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:51.371 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:51.371 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:51.371 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:51.371 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:51.371 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:51.371 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:51.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:51.705 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:51.706 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2402,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:51.723 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:51.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:52.009 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:52.026 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1644,"stamp":"2024-04-06T08:33:51.220969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.66200000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:55.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:55.338 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:55.356 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-04-06T08:33:55.222128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.76900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:55.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:55.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:55.633 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:55.649 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-04-06T08:33:55.222128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.76900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:55.650 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:55.651 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:55.651 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:55.651 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:55.651 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:55.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:33:55.976 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:55.976 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2407,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:33:55.993 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:33:55.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:56.287 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:56.306 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-04-06T08:33:55.222128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.76900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:56.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2024-04-06T08:33:59.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:59.604 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:59.604 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:59.621 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-04-06T08:33:59.223476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:59.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:33:59.908 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:33:59.909 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:33:59.925 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-04-06T08:33:59.223476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:33:59.926 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:59.927 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:33:59.927 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:59.927 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:33:59.927 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:33:59.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:00.252 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:00.252 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2411,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:00.269 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:00.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:00.545 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:00.545 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:00.561 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-04-06T08:33:59.223476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:01.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2024-04-06T08:34:03.562 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:03.855 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:03.855 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:03.871 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-04-06T08:34:03.224885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003915"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:03.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:04.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:04.142 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:04.159 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-04-06T08:34:03.224885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003915"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:04.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:04.161 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:04.161 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:04.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:04.485 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:04.485 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2415,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:04.503 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:04.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:04.767 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:04.768 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:04.784 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-04-06T08:34:03.224885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003915"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.71299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:07.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2024-04-06T08:34:07.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:08.072 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:08.073 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:08.089 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-04-06T08:34:07.226410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.66900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.309},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:08.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:08.378 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:08.378 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:08.396 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-04-06T08:34:07.226410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.66900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.309},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:08.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:08.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:08.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:08.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:08.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:08.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:08.398 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:08.398 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:08.398 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:08.398 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:08.398 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:08.398 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 596, 'reported_epoch': 449, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.260691+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.260691+0000', 'last_peered': '2024-04-06T08:33:00.260691+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.260691+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:08.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:08.727 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:08.727 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2419,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:08.744 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:08.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:09.017 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:09.017 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:09.034 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-04-06T08:34:07.226410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":596,"reported_epoch":449,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.260691+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.260691+0000","last_peered":"2024-04-06T08:33:00.260691+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.260691+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.66900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.309},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:12.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:12.338 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:12.356 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-04-06T08:34:11.227978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004502"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.623},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.70199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:12.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:12.650 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:12.650 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:12.668 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-04-06T08:34:11.227978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004502"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.623},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.70199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:12.669 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:12.669 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:12.669 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:12.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:12.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:12.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2024-04-06T08:34:13.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:13.007 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2424,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:13.024 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:13.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:13.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:13.312 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:13.330 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2024-04-06T08:34:13.228714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004580"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":463,"reported_epoch":449,"state":"active+clean","last_fresh":"2024-04-06T08:32:58.767486+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:32:58.767486+0000","last_peered":"2024-04-06T08:32:58.767486+0000","last_clean":"2024-04-06T08:32:58.767486+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:32:58.767486+0000","last_undegraded":"2024-04-06T08:32:58.767486+0000","last_fullsized":"2024-04-06T08:32:58.767486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.623},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.70199999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.45100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:16.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:16.634 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:16.634 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:16.652 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-04-06T08:34:15.229453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004567"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.51200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.495},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.41599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:16.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:16.947 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:16.947 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:16.965 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-04-06T08:34:15.229453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004567"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.51200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.495},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.41599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:16.966 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:16.966 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:16.966 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:16.967 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:16.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:17.297 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:17.297 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2428,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:17.315 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:17.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:17.593 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:17.594 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:17.610 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-04-06T08:34:17.230220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004585"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.51200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.495},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.41599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:18.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2024-04-06T08:34:20.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:20.913 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:20.914 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:20.932 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-04-06T08:34:19.230958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.745}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:20.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:21.226 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:21.226 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:21.244 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-04-06T08:34:19.230958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.745}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:21.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:21.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:21.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:21.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:21.246 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:21.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:21.583 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:21.584 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2432,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:21.602 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:21.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:21.876 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:21.892 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2024-04-06T08:34:21.231727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004546"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.745}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:23.595 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2024-04-06T08:34:24.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:25.192 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:25.193 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:25.209 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2024-04-06T08:34:23.232493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.53500000000000003,"15min":0.55700000000000005},"min":{"1min":0.152,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.65600000000000003,"5min":5.8970000000000002,"15min":11.275},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.73199999999999998,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.745}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:25.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:25.499 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:25.499 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:25.516 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-04-06T08:34:25.233242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.60599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738860,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:25.517 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:25.518 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 449, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.279831+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:00.279831+0000', 'last_peered': '2024-04-06T08:33:00.279831+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:00.279831+0000', 'last_undegraded': '2024-04-06T08:33:00.279831+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:25.518 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:25.518 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:25.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:25.847 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:25.847 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2437,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:25.864 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:25.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:26.145 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:26.145 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:26.161 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-04-06T08:34:25.233242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":449,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.279831+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:00.279831+0000","last_peered":"2024-04-06T08:33:00.279831+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:00.279831+0000","last_undegraded":"2024-04-06T08:33:00.279831+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.60599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738860,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53200000000000003,"15min":0.60799999999999998},"min":{"1min":0.20499999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.65700000000000003,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.83299999999999996,"15min":0.76600000000000001},"min":{"1min":0.26800000000000002,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.5190000000000001,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:28.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2024-04-06T08:34:29.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:29.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:29.468 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:29.486 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-04-06T08:34:29.234765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:29.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:29.781 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:29.782 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:29.798 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-04-06T08:34:29.234765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:29.799 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:29.799 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:29.800 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:29.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:30.145 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:30.145 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2441,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:30.163 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:30.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:30.432 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:30.432 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:30.449 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-04-06T08:34:29.234765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:33.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:33.742 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:33.759 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-04-06T08:34:33.236263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:33.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:34.050 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:34.051 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:34.068 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-04-06T08:34:33.236263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:34.069 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:34.069 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:34.070 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:34.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:34.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2024-04-06T08:34:34.403 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:34.403 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2445,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:34.420 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:34.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:34.684 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:34.701 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-04-06T08:34:33.236263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004536"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.50700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:37.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:37.998 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:37.999 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:38.015 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-04-06T08:34:37.237643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004401"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.65900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:38.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:38.288 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:38.288 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:38.305 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-04-06T08:34:37.237643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004401"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.65900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:38.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:38.307 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:38.307 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:38.307 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:38.307 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:38.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:38.647 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:38.647 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2449,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:38.665 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:38.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:38.936 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:38.936 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:38.952 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-04-06T08:34:37.237643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004401"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.65900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:39.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2024-04-06T08:34:41.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:42.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:42.256 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:42.274 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-04-06T08:34:41.239104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:42.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:42.562 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:42.563 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:42.579 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-04-06T08:34:41.239104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:42.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:42.581 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:42.581 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:42.581 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:42.581 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:42.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:42.930 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2454,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:42.949 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:42.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:43.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:43.241 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-04-06T08:34:41.239104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:45.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2024-04-06T08:34:46.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:46.530 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:46.546 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2024-04-06T08:34:45.240423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004159"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.60899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:46.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:46.811 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:46.811 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:46.827 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2024-04-06T08:34:45.240423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004159"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.60899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:46.828 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:46.828 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:46.828 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:46.828 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:46.829 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:46.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:47.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:47.142 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2458,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:47.159 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:47.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:47.431 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:47.431 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:47.448 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2024-04-06T08:34:47.241193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738862,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.60899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:50.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:50.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2024-04-06T08:34:50.746 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:50.747 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:50.765 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-04-06T08:34:49.241870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004227"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.497}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:50.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:51.059 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:51.060 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:51.077 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-04-06T08:34:49.241870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004227"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.497}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:51.078 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:51.079 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:51.080 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:51.080 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:51.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:51.418 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:51.419 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2462,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:51.437 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:51.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:51.718 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:51.719 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:51.735 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2024-04-06T08:34:51.242634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004298"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.497}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:54.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:55.030 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:55.031 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:55.047 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1675,"stamp":"2024-04-06T08:34:53.243239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.497}]}]},{"osd":1,"up_from":8,"seq":34359738865,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:55.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:55.341 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:55.342 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:55.360 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-04-06T08:34:55.243933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004138"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738864,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:55.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:55.362 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:55.362 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:55.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:55.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:55.708 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2466,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:34:55.727 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:34:55.727 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:55.999 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:55.999 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:56.016 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-04-06T08:34:55.243933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004138"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738864,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.42199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:56.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2024-04-06T08:34:59.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:59.324 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:59.325 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:59.343 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-04-06T08:34:59.245372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004179"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:59.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:34:59.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:59.633 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:34:59.651 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-04-06T08:34:59.245372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004179"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:34:59.652 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:34:59.652 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:34:59.653 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:34:59.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:34:59.989 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:34:59.989 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2471,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:35:00.007 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:35:00.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:00.274 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:00.289 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-04-06T08:34:59.245372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004179"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:01.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2024-04-06T08:35:03.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:03.582 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:03.583 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:03.599 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-04-06T08:35:03.246702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:03.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:03.883 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:03.883 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:03.902 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-04-06T08:35:03.246702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:35:03.903 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:03.904 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:35:03.904 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:03.904 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:35:03.904 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:03.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:35:04.256 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:04.256 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2475,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:35:04.276 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:35:04.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:04.544 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:04.544 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:04.561 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-04-06T08:35:03.246702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004068"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:06.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2024-04-06T08:35:07.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:07.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-06T08:35:07.902+0000 7f35c2ffd640 0 --2- 172.21.15.129:0/362657181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c406bb60 0x7f35c40a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-06T08:35:08.081 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:08.081 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:08.100 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1682,"stamp":"2024-04-06T08:35:07.248247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004314"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.53300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:08.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:08.387 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:08.388 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:08.405 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1682,"stamp":"2024-04-06T08:35:07.248247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004314"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.53300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:08.406 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:35:08.406 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:08.406 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:35:08.406 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:08.406 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:35:08.407 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:08.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:35:08.735 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:08.735 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2479,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:35:08.752 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:35:08.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:09.032 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:09.033 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:09.051 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1682,"stamp":"2024-04-06T08:35:07.248247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004314"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.53300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:12.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:12.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2024-04-06T08:35:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:12.352 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:12.370 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1684,"stamp":"2024-04-06T08:35:11.249786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004414"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.45000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:12.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:12.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:12.653 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:12.669 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1684,"stamp":"2024-04-06T08:35:11.249786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004414"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.45000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:12.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:35:12.670 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:12.670 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.053180+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:01.053180+0000', 'last_peered': '2024-04-06T08:33:01.053180+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:01.053180+0000', 'last_undegraded': '2024-04-06T08:33:01.053180+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:35:12.671 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:12.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:35:12.997 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:12.997 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2484,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:35:13.015 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:35:13.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:13.283 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:13.284 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:13.299 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1685,"stamp":"2024-04-06T08:35:13.250509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004485"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.053180+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:01.053180+0000","last_peered":"2024-04-06T08:33:01.053180+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:01.053180+0000","last_undegraded":"2024-04-06T08:33:01.053180+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.45000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:16.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:16.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:16.597 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:16.612 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2024-04-06T08:35:15.251239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004537"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:04.082341+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:04.082341+0000","last_peered":"2024-04-06T08:33:04.082341+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:04.082341+0000","last_undegraded":"2024-04-06T08:33:04.082341+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.56599999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.434},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.46500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:16.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:16.888 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:16.888 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:16.904 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2024-04-06T08:35:15.251239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004537"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:04.082341+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:04.082341+0000","last_peered":"2024-04-06T08:33:04.082341+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:04.082341+0000","last_undegraded":"2024-04-06T08:33:04.082341+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.56599999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.434},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.46500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:16.905 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-04-06T08:35:16.905 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.994417+0000', 'last_change': '2024-04-06T08:30:16.208944+0000', 'last_active': '2024-04-06T08:33:00.994417+0000', 'last_peered': '2024-04-06T08:33:00.994417+0000', 'last_clean': '2024-04-06T08:30:07.737055+0000', 'last_became_active': '2024-04-06T08:30:16.208944+0000', 'last_became_peered': '2024-04-06T08:30:16.208944+0000', 'last_unstale': '2024-04-06T08:33:00.994417+0000', 'last_undegraded': '2024-04-06T08:33:00.994417+0000', 'last_fullsized': '2024-04-06T08:30:16.195169+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:16.905 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-04-06T08:35:16.905 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:04.082341+0000', 'last_change': '2024-04-06T08:30:16.214210+0000', 'last_active': '2024-04-06T08:33:04.082341+0000', 'last_peered': '2024-04-06T08:33:04.082341+0000', 'last_clean': '2024-04-06T08:30:07.736978+0000', 'last_became_active': '2024-04-06T08:30:16.214210+0000', 'last_became_peered': '2024-04-06T08:30:16.214210+0000', 'last_unstale': '2024-04-06T08:33:04.082341+0000', 'last_undegraded': '2024-04-06T08:33:04.082341+0000', 'last_fullsized': '2024-04-06T08:30:16.195944+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:16.905 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-04-06T08:35:16.905 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:00.914795+0000', 'last_change': '2024-04-06T08:30:16.216428+0000', 'last_active': '2024-04-06T08:33:00.914795+0000', 'last_peered': '2024-04-06T08:33:00.914795+0000', 'last_clean': '2024-04-06T08:30:07.736206+0000', 'last_became_active': '2024-04-06T08:30:16.216428+0000', 'last_became_peered': '2024-04-06T08:30:16.216428+0000', 'last_unstale': '2024-04-06T08:33:00.914795+0000', 'last_undegraded': '2024-04-06T08:33:00.914795+0000', 'last_fullsized': '2024-04-06T08:30:16.201317+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:16.906 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-04-06T08:35:16.906 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 466, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.283107+0000', 'last_change': '2024-04-06T08:30:16.219424+0000', 'last_active': '2024-04-06T08:33:01.283107+0000', 'last_peered': '2024-04-06T08:33:01.283107+0000', 'last_clean': '2024-04-06T08:30:07.734225+0000', 'last_became_active': '2024-04-06T08:30:16.219424+0000', 'last_became_peered': '2024-04-06T08:30:16.219424+0000', 'last_unstale': '2024-04-06T08:33:01.283107+0000', 'last_undegraded': '2024-04-06T08:33:01.283107+0000', 'last_fullsized': '2024-04-06T08:30:16.184711+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:16.906 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-04-06T08:35:16.906 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 450, 'state': 'active+undersized', 'last_fresh': '2024-04-06T08:33:01.176516+0000', 'last_change': '2024-04-06T08:30:16.231848+0000', 'last_active': '2024-04-06T08:33:01.176516+0000', 'last_peered': '2024-04-06T08:33:01.176516+0000', 'last_clean': '2024-04-06T07:54:04.757629+0000', 'last_became_active': '2024-04-06T08:30:16.231848+0000', 'last_became_peered': '2024-04-06T08:30:16.231848+0000', 'last_unstale': '2024-04-06T08:33:01.176516+0000', 'last_undegraded': '2024-04-06T08:33:01.176516+0000', 'last_fullsized': '2024-04-06T08:30:16.195208+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'last_clean_scrub_stamp': '2024-04-06T07:54:04.757629+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:16.906 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-04-06T08:35:16.906 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 597, 'reported_epoch': 450, 'state': 'active+undersized+degraded', 'last_fresh': '2024-04-06T08:33:00.795171+0000', 'last_change': '2024-04-06T08:30:16.205073+0000', 'last_active': '2024-04-06T08:33:00.795171+0000', 'last_peered': '2024-04-06T08:33:00.795171+0000', 'last_clean': '2024-04-06T08:30:08.986522+0000', 'last_became_active': '2024-04-06T08:30:16.205073+0000', 'last_became_peered': '2024-04-06T08:30:16.205073+0000', 'last_unstale': '2024-04-06T08:33:00.795171+0000', 'last_undegraded': '2024-04-06T08:30:16.182433+0000', 'last_fullsized': '2024-04-06T08:30:16.177233+0000', 'mapping_epoch': 436, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'last_clean_scrub_stamp': '2024-04-06T07:53:59.281890+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-06T08:35:16.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-04-06T08:35:17.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:17.236 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"fd3eff46-be4e-471a-92ae-a94b26dd3208","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2488,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":450,"num_osds":3,"num_up_osds":2,"osd_up_since":1712392215,"num_in_osds":3,"osd_in_since":1712390029,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3687948288,"bytes_avail":286222344192,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":48,"modified":"2024-04-06T08:33:09.208243+0000","services":{}},"progress_events":{}} 2024-04-06T08:35:17.253 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-04-06T08:35:17.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:35:17.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-06T08:35:17.541 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-06T08:35:17.559 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":1687,"stamp":"2024-04-06T08:35:17.252038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3601512,"kb_used_data":365580,"kb_used_omap":292,"kb_used_meta":3211483,"kb_avail":279514008,"statfs":{"total":289910292480,"available":286222344192,"internally_reserved":0,"allocated":374353920,"data_stored":365956672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299082,"internal_metadata":3288559542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004579"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":465,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.994417+0000","last_change":"2024-04-06T08:30:16.208944+0000","last_active":"2024-04-06T08:33:00.994417+0000","last_peered":"2024-04-06T08:33:00.994417+0000","last_clean":"2024-04-06T08:30:07.737055+0000","last_became_active":"2024-04-06T08:30:16.208944+0000","last_became_peered":"2024-04-06T08:30:16.208944+0000","last_unstale":"2024-04-06T08:33:00.994417+0000","last_undegraded":"2024-04-06T08:33:00.994417+0000","last_fullsized":"2024-04-06T08:30:16.195169+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:03:27.692930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:04.082341+0000","last_change":"2024-04-06T08:30:16.214210+0000","last_active":"2024-04-06T08:33:04.082341+0000","last_peered":"2024-04-06T08:33:04.082341+0000","last_clean":"2024-04-06T08:30:07.736978+0000","last_became_active":"2024-04-06T08:30:16.214210+0000","last_became_peered":"2024-04-06T08:30:16.214210+0000","last_unstale":"2024-04-06T08:33:04.082341+0000","last_undegraded":"2024-04-06T08:33:04.082341+0000","last_fullsized":"2024-04-06T08:30:16.195944+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:00:32.010664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:00.914795+0000","last_change":"2024-04-06T08:30:16.216428+0000","last_active":"2024-04-06T08:33:00.914795+0000","last_peered":"2024-04-06T08:33:00.914795+0000","last_clean":"2024-04-06T08:30:07.736206+0000","last_became_active":"2024-04-06T08:30:16.216428+0000","last_became_peered":"2024-04-06T08:30:16.216428+0000","last_unstale":"2024-04-06T08:33:00.914795+0000","last_undegraded":"2024-04-06T08:33:00.914795+0000","last_fullsized":"2024-04-06T08:30:16.201317+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:37:15.509358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":466,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.283107+0000","last_change":"2024-04-06T08:30:16.219424+0000","last_active":"2024-04-06T08:33:01.283107+0000","last_peered":"2024-04-06T08:33:01.283107+0000","last_clean":"2024-04-06T08:30:07.734225+0000","last_became_active":"2024-04-06T08:30:16.219424+0000","last_became_peered":"2024-04-06T08:30:16.219424+0000","last_unstale":"2024-04-06T08:33:01.283107+0000","last_undegraded":"2024-04-06T08:33:01.283107+0000","last_fullsized":"2024-04-06T08:30:16.184711+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:21:17.198849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+undersized","last_fresh":"2024-04-06T08:33:01.176516+0000","last_change":"2024-04-06T08:30:16.231848+0000","last_active":"2024-04-06T08:33:01.176516+0000","last_peered":"2024-04-06T08:33:01.176516+0000","last_clean":"2024-04-06T07:54:04.757629+0000","last_became_active":"2024-04-06T08:30:16.231848+0000","last_became_peered":"2024-04-06T08:30:16.231848+0000","last_unstale":"2024-04-06T08:33:01.176516+0000","last_undegraded":"2024-04-06T08:33:01.176516+0000","last_fullsized":"2024-04-06T08:30:16.195208+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:14:22.888239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.491514+0000","last_change":"2024-04-06T07:54:05.784303+0000","last_active":"2024-04-06T08:33:01.491514+0000","last_peered":"2024-04-06T08:33:01.491514+0000","last_clean":"2024-04-06T08:33:01.491514+0000","last_became_active":"2024-04-06T07:54:05.783541+0000","last_became_peered":"2024-04-06T07:54:05.783541+0000","last_unstale":"2024-04-06T08:33:01.491514+0000","last_undegraded":"2024-04-06T08:33:01.491514+0000","last_fullsized":"2024-04-06T08:33:01.491514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:16:02.906867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:01.492144+0000","last_change":"2024-04-06T07:54:05.785533+0000","last_active":"2024-04-06T08:33:01.492144+0000","last_peered":"2024-04-06T08:33:01.492144+0000","last_clean":"2024-04-06T08:33:01.492144+0000","last_became_active":"2024-04-06T07:54:05.784972+0000","last_became_peered":"2024-04-06T07:54:05.784972+0000","last_unstale":"2024-04-06T08:33:01.492144+0000","last_undegraded":"2024-04-06T08:33:01.492144+0000","last_fullsized":"2024-04-06T08:33:01.492144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:35:44.825543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":461,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-04-06T08:33:00.873616+0000","last_change":"2024-04-06T07:54:05.781514+0000","last_active":"2024-04-06T08:33:00.873616+0000","last_peered":"2024-04-06T08:33:00.873616+0000","last_clean":"2024-04-06T08:33:00.873616+0000","last_became_active":"2024-04-06T07:54:05.780710+0000","last_became_peered":"2024-04-06T07:54:05.780710+0000","last_unstale":"2024-04-06T08:33:00.873616+0000","last_undegraded":"2024-04-06T08:33:00.873616+0000","last_fullsized":"2024-04-06T08:33:00.873616+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:54:04.757629+0000","last_clean_scrub_stamp":"2024-04-06T07:54:04.757629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T08:29:01.689045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":597,"reported_epoch":450,"state":"active+undersized+degraded","last_fresh":"2024-04-06T08:33:00.795171+0000","last_change":"2024-04-06T08:30:16.205073+0000","last_active":"2024-04-06T08:33:00.795171+0000","last_peered":"2024-04-06T08:33:00.795171+0000","last_clean":"2024-04-06T08:30:08.986522+0000","last_became_active":"2024-04-06T08:30:16.205073+0000","last_became_peered":"2024-04-06T08:30:16.205073+0000","last_unstale":"2024-04-06T08:33:00.795171+0000","last_undegraded":"2024-04-06T08:30:16.182433+0000","last_fullsized":"2024-04-06T08:30:16.177233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:59.281890+0000","last_clean_scrub_stamp":"2024-04-06T07:53:59.281890+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:36:47.340863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738868,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":882600,"kb_used_data":5212,"kb_used_omap":99,"kb_used_meta":877276,"kb_avail":93489240,"statfs":{"total":96636764160,"available":95732981760,"internally_reserved":0,"allocated":5337088,"data_stored":2615140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101579,"internal_metadata":898331445},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:30:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.753,"15min":0.63200000000000001},"min":{"1min":0.16500000000000001,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":5.141,"5min":9.1180000000000003,"15min":9.1180000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64400000000000002,"15min":0.68100000000000005},"min":{"1min":0.19500000000000001,"5min":0.16,"15min":0.16},"max":{"1min":1.859,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.183}]},{"osd":1,"last update":"Sat Apr 6 08:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.224,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.67900000000000005,"5min":5.8970000000000002,"15min":5.8970000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.622},"min":{"1min":0.28000000000000003,"5min":0.19,"15min":0.19},"max":{"1min":0.76900000000000002,"5min":5.9340000000000002,"15min":5.9340000000000002},"last":0.56599999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1082352,"kb_used_data":5792,"kb_used_omap":86,"kb_used_meta":1076457,"kb_avail":93289488,"statfs":{"total":96636764160,"available":95528435712,"internally_reserved":0,"allocated":5931008,"data_stored":3205508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88706,"internal_metadata":1102292350},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.92000000000000004,"15min":0.69799999999999995},"min":{"1min":0.20799999999999999,"5min":0.19500000000000001,"15min":0.16800000000000001},"max":{"1min":2.621,"5min":15.191000000000001,"15min":15.191000000000001},"last":0.315},{"interface":"front","average":{"1min":0.502,"5min":0.745,"15min":0.621},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":12.5,"15min":12.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Apr 6 08:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.51400000000000001,"15min":0.58499999999999996},"min":{"1min":0.25600000000000001,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.55100000000000005,"5min":2.9009999999999998,"15min":12.539999999999999},"last":0.434},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.23300000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.56100000000000005,"5min":14.470000000000001,"15min":14.470000000000001},"last":0.46500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1636560,"kb_used_data":354576,"kb_used_omap":106,"kb_used_meta":1257749,"kb_avail":92735280,"statfs":{"total":96636764160,"available":94960926720,"internally_reserved":0,"allocated":363085824,"data_stored":360136024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108797,"internal_metadata":1287935747},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.20300000000000001,"5min":0.16,"15min":0.158},"max":{"1min":1.3180000000000001,"5min":4.5380000000000003,"15min":8.048},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.47699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.183,"15min":0.14699999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":16.041},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 6 08:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.60799999999999998,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":2.1970000000000001,"5min":4.6029999999999998,"15min":9.7789999999999999},"last":1.79},{"interface":"front","average":{"1min":0.623,"5min":0.64000000000000001,"15min":0.71699999999999997},"min":{"1min":0.28199999999999997,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":0.76500000000000001,"5min":10.07,"15min":10.07},"last":1.7749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:35:17.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2024-04-06T08:35:17.635 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-06T08:35:18.039 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2024-04-06T08:35:18.039 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2024-04-06T08:35:18.040 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2024-04-06T08:35:18.040 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2024-04-06T08:35:18.040 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T08:35:18.042+0000 7fcd3ed21640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 23996) UID: 0 2024-04-06T08:35:18.040 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T08:35:18.042+0000 7fcd3ed21640 -1 osd.1 450 *** Got signal Terminated *** 2024-04-06T08:35:18.041 INFO:tasks.ceph.cluster2.osd.1.smithi129.stderr:2024-04-06T08:35:18.042+0000 7fcd3ed21640 -1 osd.1 450 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T08:35:18.041 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:35:18.042+0000 7f5a226f5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 23999) UID: 0 2024-04-06T08:35:18.041 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:35:18.042+0000 7f5a226f5640 -1 osd.2 450 *** Got signal Terminated *** 2024-04-06T08:35:18.041 INFO:tasks.ceph.cluster2.osd.2.smithi129.stderr:2024-04-06T08:35:18.042+0000 7f5a226f5640 -1 osd.2 450 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T08:35:18.041 INFO:tasks.ceph.cluster2.mon.a.smithi129.stderr:2024-04-06T08:35:18.042+0000 7f914d63e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 23848) UID: 0 2024-04-06T08:35:18.041 INFO:tasks.ceph.cluster2.mon.a.smithi129.stderr:2024-04-06T08:35:18.042+0000 7f914d63e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-06T08:35:18.042 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T08:35:18.038+0000 7fe5a4b57640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 28014) UID: 0 2024-04-06T08:35:18.042 INFO:tasks.ceph.cluster2.mgr.x.smithi047.stderr:2024-04-06T08:35:18.038+0000 7fe5a4b57640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-06T08:35:20.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-06T08:37:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-06T08:37:20.583 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2765, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2701, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-06T08:37:20.585 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-06T08:37:20.586 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-06T08:37:20.586 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-06T08:37:20.586 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:37:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-04-06T08:37:20.586 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2765, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2701, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2024-04-06T08:37:20.587 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-06T08:37:20.587 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-06T08:37:20.587 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:37:20.587 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-06T08:37:20.587 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-06T08:37:20.587 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:37:20.588 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-06T08:37:20.588 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-06T08:37:20.588 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-06T08:37:20.588 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:37:20.588 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-06T08:37:20.588 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-06T08:37:20.588 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-06T08:37:20.588 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:37:20.588 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-06T08:37:20.589 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-06T08:37:20.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T08:37:20.603 INFO:teuthology.orchestra.run.smithi129.stdout:2024-04-06T08:30:15.161869+0000 mon.a (mon.0) 1591 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T08:37:20.604 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-06T08:37:20.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T08:37:20.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T08:37:20.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[WRN\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T08:37:20.771 INFO:teuthology.orchestra.run.smithi129.stdout:2024-04-06T08:30:15.161869+0000 mon.a (mon.0) 1591 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-06T08:37:20.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T08:37:20.771 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-06T08:37:21.195 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T08:37:21.195 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-06T08:37:21.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T08:37:21.541 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-06T08:37:21.826 INFO:tasks.ceph:Archiving mon data... 2024-04-06T08:37:21.828 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/data/cluster2.mon.a.tgz 2024-04-06T08:37:21.828 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-04-06T08:37:21.833 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.ozApSfVk6U 2024-04-06T08:37:21.834 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.ozApSfVk6U 2024-04-06T08:37:21.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.ozApSfVk6U 2024-04-06T08:37:22.047 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.ozApSfVk6U is 471KB 2024-04-06T08:37:22.109 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.ozApSfVk6U 2024-04-06T08:37:22.114 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-06T08:37:22.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T08:37:22.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T08:37:22.171 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2765, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2701, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-06T08:37:22.172 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-06T08:37:22.189 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-06T08:37:22.189 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T08:37:22.493 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:37:22.493 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T08:37:22.511 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1785,"stamp":"2024-04-06T08:37:22.171398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312292,"kb_used_data":12416,"kb_used_omap":4,"kb_used_meta":299835,"kb_avail":282803228,"statfs":{"total":289910292480,"available":289590505472,"internally_reserved":0,"allocated":12713984,"data_stored":7374506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":307031378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:43.148436+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T08:34:43.148436+0000","last_peered":"2024-04-06T08:34:43.148436+0000","last_clean":"2024-04-06T08:34:43.148436+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T08:34:43.148436+0000","last_undegraded":"2024-04-06T08:34:43.148436+0000","last_fullsized":"2024-04-06T08:34:43.148436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:44.700586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:38.147644+0000","last_change":"2024-04-06T07:53:29.233161+0000","last_active":"2024-04-06T08:34:38.147644+0000","last_peered":"2024-04-06T08:34:38.147644+0000","last_clean":"2024-04-06T08:34:38.147644+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T08:34:38.147644+0000","last_undegraded":"2024-04-06T08:34:38.147644+0000","last_fullsized":"2024-04-06T08:34:38.147644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:34.387543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152927+0000","last_change":"2024-04-06T07:53:29.235401+0000","last_active":"2024-04-06T08:35:13.152927+0000","last_peered":"2024-04-06T08:35:13.152927+0000","last_clean":"2024-04-06T08:35:13.152927+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T08:35:13.152927+0000","last_undegraded":"2024-04-06T08:35:13.152927+0000","last_fullsized":"2024-04-06T08:35:13.152927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:05.919002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152899+0000","last_change":"2024-04-06T07:53:29.233180+0000","last_active":"2024-04-06T08:35:13.152899+0000","last_peered":"2024-04-06T08:35:13.152899+0000","last_clean":"2024-04-06T08:35:13.152899+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T08:35:13.152899+0000","last_undegraded":"2024-04-06T08:35:13.152899+0000","last_fullsized":"2024-04-06T08:35:13.152899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:51:57.674391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:12.501017+0000","last_change":"2024-04-06T07:53:29.231074+0000","last_active":"2024-04-06T08:35:12.501017+0000","last_peered":"2024-04-06T08:35:12.501017+0000","last_clean":"2024-04-06T08:35:12.501017+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T08:35:12.501017+0000","last_undegraded":"2024-04-06T08:35:12.501017+0000","last_fullsized":"2024-04-06T08:35:12.501017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:19:19.458982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:07.459490+0000","last_change":"2024-04-06T07:53:29.233823+0000","last_active":"2024-04-06T08:35:07.459490+0000","last_peered":"2024-04-06T08:35:07.459490+0000","last_clean":"2024-04-06T08:35:07.459490+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T08:35:07.459490+0000","last_undegraded":"2024-04-06T08:35:07.459490+0000","last_fullsized":"2024-04-06T08:35:07.459490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:37.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:42.447789+0000","last_change":"2024-04-06T07:53:29.231274+0000","last_active":"2024-04-06T08:33:42.447789+0000","last_peered":"2024-04-06T08:33:42.447789+0000","last_clean":"2024-04-06T08:33:42.447789+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T08:33:42.447789+0000","last_undegraded":"2024-04-06T08:33:42.447789+0000","last_fullsized":"2024-04-06T08:33:42.447789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:24.362133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152872+0000","last_change":"2024-04-06T07:53:29.232232+0000","last_active":"2024-04-06T08:35:13.152872+0000","last_peered":"2024-04-06T08:35:13.152872+0000","last_clean":"2024-04-06T08:35:13.152872+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T08:35:13.152872+0000","last_undegraded":"2024-04-06T08:35:13.152872+0000","last_fullsized":"2024-04-06T08:35:13.152872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:23.930116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":613,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:28.137100+0000","last_change":"2024-04-06T07:53:24.763344+0000","last_active":"2024-04-06T08:33:28.137100+0000","last_peered":"2024-04-06T08:33:28.137100+0000","last_clean":"2024-04-06T08:33:28.137100+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T08:33:28.137100+0000","last_undegraded":"2024-04-06T08:33:28.137100+0000","last_fullsized":"2024-04-06T08:33:28.137100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_clean_scrub_stamp":"2024-04-06T07:53:23.707131+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:54.646711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97140,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274700,"statfs":{"total":96636764160,"available":96537292800,"internally_reserved":0,"allocated":3842048,"data_stored":2064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.39900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76800000000000002,"15min":2.661},"last":0.437},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63900000000000001,"15min":0.67700000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":13.782999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.374},"max":{"1min":0.63600000000000001,"5min":0.76100000000000001,"15min":43.079999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.621},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.436},"max":{"1min":0.73599999999999999,"5min":0.754,"15min":1.3500000000000001},"last":0.58099999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738900,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89528,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94282312,"statfs":{"total":96636764160,"available":96545087488,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.30299999999999999},"max":{"1min":0.67400000000000004,"5min":0.73199999999999998,"15min":6.7370000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":14.582000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46100000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.88800000000000001,"15min":2.931},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.53300000000000003,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72399999999999998,"5min":0.96599999999999997,"15min":1.288},"last":0.751}]}]},{"osd":0,"up_from":8,"seq":34359738899,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125624,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":121278,"kb_avail":94246216,"statfs":{"total":96636764160,"available":96508125184,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124189126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.25600000000000001},"max":{"1min":0.63800000000000001,"5min":1.004,"15min":16.492999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.61399999999999999},"min":{"1min":0.52800000000000002,"5min":0.44,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":1.601,"15min":5.6680000000000001},"last":0.58999999999999997}]},{"osd":2,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.71899999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":4.0430000000000001,"15min":10.403},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":4.7320000000000002,"15min":6.5270000000000001},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:37:22.513 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T08:37:22.811 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:37:22.811 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T08:37:22.829 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1785,"stamp":"2024-04-06T08:37:22.171398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312292,"kb_used_data":12416,"kb_used_omap":4,"kb_used_meta":299835,"kb_avail":282803228,"statfs":{"total":289910292480,"available":289590505472,"internally_reserved":0,"allocated":12713984,"data_stored":7374506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":307031378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:43.148436+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T08:34:43.148436+0000","last_peered":"2024-04-06T08:34:43.148436+0000","last_clean":"2024-04-06T08:34:43.148436+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T08:34:43.148436+0000","last_undegraded":"2024-04-06T08:34:43.148436+0000","last_fullsized":"2024-04-06T08:34:43.148436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:44.700586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:38.147644+0000","last_change":"2024-04-06T07:53:29.233161+0000","last_active":"2024-04-06T08:34:38.147644+0000","last_peered":"2024-04-06T08:34:38.147644+0000","last_clean":"2024-04-06T08:34:38.147644+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T08:34:38.147644+0000","last_undegraded":"2024-04-06T08:34:38.147644+0000","last_fullsized":"2024-04-06T08:34:38.147644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:34.387543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152927+0000","last_change":"2024-04-06T07:53:29.235401+0000","last_active":"2024-04-06T08:35:13.152927+0000","last_peered":"2024-04-06T08:35:13.152927+0000","last_clean":"2024-04-06T08:35:13.152927+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T08:35:13.152927+0000","last_undegraded":"2024-04-06T08:35:13.152927+0000","last_fullsized":"2024-04-06T08:35:13.152927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:05.919002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152899+0000","last_change":"2024-04-06T07:53:29.233180+0000","last_active":"2024-04-06T08:35:13.152899+0000","last_peered":"2024-04-06T08:35:13.152899+0000","last_clean":"2024-04-06T08:35:13.152899+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T08:35:13.152899+0000","last_undegraded":"2024-04-06T08:35:13.152899+0000","last_fullsized":"2024-04-06T08:35:13.152899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:51:57.674391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:12.501017+0000","last_change":"2024-04-06T07:53:29.231074+0000","last_active":"2024-04-06T08:35:12.501017+0000","last_peered":"2024-04-06T08:35:12.501017+0000","last_clean":"2024-04-06T08:35:12.501017+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T08:35:12.501017+0000","last_undegraded":"2024-04-06T08:35:12.501017+0000","last_fullsized":"2024-04-06T08:35:12.501017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:19:19.458982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:07.459490+0000","last_change":"2024-04-06T07:53:29.233823+0000","last_active":"2024-04-06T08:35:07.459490+0000","last_peered":"2024-04-06T08:35:07.459490+0000","last_clean":"2024-04-06T08:35:07.459490+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T08:35:07.459490+0000","last_undegraded":"2024-04-06T08:35:07.459490+0000","last_fullsized":"2024-04-06T08:35:07.459490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:37.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:42.447789+0000","last_change":"2024-04-06T07:53:29.231274+0000","last_active":"2024-04-06T08:33:42.447789+0000","last_peered":"2024-04-06T08:33:42.447789+0000","last_clean":"2024-04-06T08:33:42.447789+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T08:33:42.447789+0000","last_undegraded":"2024-04-06T08:33:42.447789+0000","last_fullsized":"2024-04-06T08:33:42.447789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:24.362133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152872+0000","last_change":"2024-04-06T07:53:29.232232+0000","last_active":"2024-04-06T08:35:13.152872+0000","last_peered":"2024-04-06T08:35:13.152872+0000","last_clean":"2024-04-06T08:35:13.152872+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T08:35:13.152872+0000","last_undegraded":"2024-04-06T08:35:13.152872+0000","last_fullsized":"2024-04-06T08:35:13.152872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:23.930116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":613,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:28.137100+0000","last_change":"2024-04-06T07:53:24.763344+0000","last_active":"2024-04-06T08:33:28.137100+0000","last_peered":"2024-04-06T08:33:28.137100+0000","last_clean":"2024-04-06T08:33:28.137100+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T08:33:28.137100+0000","last_undegraded":"2024-04-06T08:33:28.137100+0000","last_fullsized":"2024-04-06T08:33:28.137100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_clean_scrub_stamp":"2024-04-06T07:53:23.707131+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:54.646711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97140,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274700,"statfs":{"total":96636764160,"available":96537292800,"internally_reserved":0,"allocated":3842048,"data_stored":2064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.39900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76800000000000002,"15min":2.661},"last":0.437},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63900000000000001,"15min":0.67700000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":13.782999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.374},"max":{"1min":0.63600000000000001,"5min":0.76100000000000001,"15min":43.079999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.621},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.436},"max":{"1min":0.73599999999999999,"5min":0.754,"15min":1.3500000000000001},"last":0.58099999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738900,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89528,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94282312,"statfs":{"total":96636764160,"available":96545087488,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.30299999999999999},"max":{"1min":0.67400000000000004,"5min":0.73199999999999998,"15min":6.7370000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":14.582000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46100000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.88800000000000001,"15min":2.931},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.53300000000000003,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72399999999999998,"5min":0.96599999999999997,"15min":1.288},"last":0.751}]}]},{"osd":0,"up_from":8,"seq":34359738899,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125624,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":121278,"kb_avail":94246216,"statfs":{"total":96636764160,"available":96508125184,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124189126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.25600000000000001},"max":{"1min":0.63800000000000001,"5min":1.004,"15min":16.492999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.61399999999999999},"min":{"1min":0.52800000000000002,"5min":0.44,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":1.601,"15min":5.6680000000000001},"last":0.58999999999999997}]},{"osd":2,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.71899999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":4.0430000000000001,"15min":10.403},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":4.7320000000000002,"15min":6.5270000000000001},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:37:22.830 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-06T08:37:22.830 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T08:37:23.115 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:37:23.115 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T08:37:23.132 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1785,"stamp":"2024-04-06T08:37:22.171398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312292,"kb_used_data":12416,"kb_used_omap":4,"kb_used_meta":299835,"kb_avail":282803228,"statfs":{"total":289910292480,"available":289590505472,"internally_reserved":0,"allocated":12713984,"data_stored":7374506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":307031378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:43.148436+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T08:34:43.148436+0000","last_peered":"2024-04-06T08:34:43.148436+0000","last_clean":"2024-04-06T08:34:43.148436+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T08:34:43.148436+0000","last_undegraded":"2024-04-06T08:34:43.148436+0000","last_fullsized":"2024-04-06T08:34:43.148436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:44.700586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:38.147644+0000","last_change":"2024-04-06T07:53:29.233161+0000","last_active":"2024-04-06T08:34:38.147644+0000","last_peered":"2024-04-06T08:34:38.147644+0000","last_clean":"2024-04-06T08:34:38.147644+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T08:34:38.147644+0000","last_undegraded":"2024-04-06T08:34:38.147644+0000","last_fullsized":"2024-04-06T08:34:38.147644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:34.387543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152927+0000","last_change":"2024-04-06T07:53:29.235401+0000","last_active":"2024-04-06T08:35:13.152927+0000","last_peered":"2024-04-06T08:35:13.152927+0000","last_clean":"2024-04-06T08:35:13.152927+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T08:35:13.152927+0000","last_undegraded":"2024-04-06T08:35:13.152927+0000","last_fullsized":"2024-04-06T08:35:13.152927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:05.919002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152899+0000","last_change":"2024-04-06T07:53:29.233180+0000","last_active":"2024-04-06T08:35:13.152899+0000","last_peered":"2024-04-06T08:35:13.152899+0000","last_clean":"2024-04-06T08:35:13.152899+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T08:35:13.152899+0000","last_undegraded":"2024-04-06T08:35:13.152899+0000","last_fullsized":"2024-04-06T08:35:13.152899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:51:57.674391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:12.501017+0000","last_change":"2024-04-06T07:53:29.231074+0000","last_active":"2024-04-06T08:35:12.501017+0000","last_peered":"2024-04-06T08:35:12.501017+0000","last_clean":"2024-04-06T08:35:12.501017+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T08:35:12.501017+0000","last_undegraded":"2024-04-06T08:35:12.501017+0000","last_fullsized":"2024-04-06T08:35:12.501017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:19:19.458982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:07.459490+0000","last_change":"2024-04-06T07:53:29.233823+0000","last_active":"2024-04-06T08:35:07.459490+0000","last_peered":"2024-04-06T08:35:07.459490+0000","last_clean":"2024-04-06T08:35:07.459490+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T08:35:07.459490+0000","last_undegraded":"2024-04-06T08:35:07.459490+0000","last_fullsized":"2024-04-06T08:35:07.459490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:37.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:42.447789+0000","last_change":"2024-04-06T07:53:29.231274+0000","last_active":"2024-04-06T08:33:42.447789+0000","last_peered":"2024-04-06T08:33:42.447789+0000","last_clean":"2024-04-06T08:33:42.447789+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T08:33:42.447789+0000","last_undegraded":"2024-04-06T08:33:42.447789+0000","last_fullsized":"2024-04-06T08:33:42.447789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:24.362133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152872+0000","last_change":"2024-04-06T07:53:29.232232+0000","last_active":"2024-04-06T08:35:13.152872+0000","last_peered":"2024-04-06T08:35:13.152872+0000","last_clean":"2024-04-06T08:35:13.152872+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T08:35:13.152872+0000","last_undegraded":"2024-04-06T08:35:13.152872+0000","last_fullsized":"2024-04-06T08:35:13.152872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:23.930116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":613,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:28.137100+0000","last_change":"2024-04-06T07:53:24.763344+0000","last_active":"2024-04-06T08:33:28.137100+0000","last_peered":"2024-04-06T08:33:28.137100+0000","last_clean":"2024-04-06T08:33:28.137100+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T08:33:28.137100+0000","last_undegraded":"2024-04-06T08:33:28.137100+0000","last_fullsized":"2024-04-06T08:33:28.137100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_clean_scrub_stamp":"2024-04-06T07:53:23.707131+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:54.646711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97140,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274700,"statfs":{"total":96636764160,"available":96537292800,"internally_reserved":0,"allocated":3842048,"data_stored":2064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.39900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76800000000000002,"15min":2.661},"last":0.437},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63900000000000001,"15min":0.67700000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":13.782999999999999},"last":0.61399999999999999}]},{"osd":1,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.374},"max":{"1min":0.63600000000000001,"5min":0.76100000000000001,"15min":43.079999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.621},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.436},"max":{"1min":0.73599999999999999,"5min":0.754,"15min":1.3500000000000001},"last":0.58099999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738900,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89528,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94282312,"statfs":{"total":96636764160,"available":96545087488,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.30299999999999999},"max":{"1min":0.67400000000000004,"5min":0.73199999999999998,"15min":6.7370000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":14.582000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46100000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.88800000000000001,"15min":2.931},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.53300000000000003,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72399999999999998,"5min":0.96599999999999997,"15min":1.288},"last":0.751}]}]},{"osd":0,"up_from":8,"seq":34359738899,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125624,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":121278,"kb_avail":94246216,"statfs":{"total":96636764160,"available":96508125184,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124189126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.25600000000000001},"max":{"1min":0.63800000000000001,"5min":1.004,"15min":16.492999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.61399999999999999},"min":{"1min":0.52800000000000002,"5min":0.44,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":1.601,"15min":5.6680000000000001},"last":0.58999999999999997}]},{"osd":2,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.71899999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":4.0430000000000001,"15min":10.403},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":4.7320000000000002,"15min":6.5270000000000001},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:37:23.133 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-06T08:37:23.646 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:37:23.646 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":467,"fsid":"b40bb857-7ded-43c7-8c1c-38011a993fe5","created":"2024-04-06T07:53:12.245738+0000","modified":"2024-04-06T08:33:00.166619+0000","last_up_change":"2024-04-06T07:53:21.704210+0000","last_in_change":"2024-04-06T07:53:13.136443+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T07:53:23.281297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-06T07:53:27.663437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c374052-d1fe-488f-878f-354cadfee4ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":23782},{"type":"v1","addr":"172.21.15.47:6817","nonce":23782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":23782},{"type":"v1","addr":"172.21.15.47:6819","nonce":23782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":23782},{"type":"v1","addr":"172.21.15.47:6823","nonce":23782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":23782},{"type":"v1","addr":"172.21.15.47:6821","nonce":23782}]},"public_addr":"172.21.15.47:6817/23782","cluster_addr":"172.21.15.47:6819/23782","heartbeat_back_addr":"172.21.15.47:6823/23782","heartbeat_front_addr":"172.21.15.47:6821/23782","state":["exists","up"]},{"osd":1,"uuid":"84cff3d1-8b34-4a58-bffc-a4cc844f7212","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":23778},{"type":"v1","addr":"172.21.15.47:6801","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":23778},{"type":"v1","addr":"172.21.15.47:6803","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":23778},{"type":"v1","addr":"172.21.15.47:6807","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":23778},{"type":"v1","addr":"172.21.15.47:6805","nonce":23778}]},"public_addr":"172.21.15.47:6801/23778","cluster_addr":"172.21.15.47:6803/23778","heartbeat_back_addr":"172.21.15.47:6807/23778","heartbeat_front_addr":"172.21.15.47:6805/23778","state":["exists","up"]},{"osd":2,"uuid":"7369452f-d483-4735-928d-aa461b4cffb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":23779},{"type":"v1","addr":"172.21.15.47:6809","nonce":23779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":23779},{"type":"v1","addr":"172.21.15.47:6811","nonce":23779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":23779},{"type":"v1","addr":"172.21.15.47:6815","nonce":23779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":23779},{"type":"v1","addr":"172.21.15.47:6813","nonce":23779}]},"public_addr":"172.21.15.47:6809/23779","cluster_addr":"172.21.15.47:6811/23779","heartbeat_back_addr":"172.21.15.47:6815/23779","heartbeat_front_addr":"172.21.15.47:6813/23779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.630582+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.635946+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T07:53:18.596933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T08:37:24.665 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-04-06T08:37:24.666 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-06T08:37:24.829 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-06T08:37:24.829 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T08:37:24.830 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-06T08:37:24.846 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-04-06T08:37:25.138 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 0 to deep-scrub 2024-04-06T08:37:25.157 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-04-06T08:37:25.157 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-06T08:37:25.309 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-06T08:37:25.309 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T08:37:25.309 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-06T08:37:25.326 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-04-06T08:37:25.614 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 1 to deep-scrub 2024-04-06T08:37:25.633 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-04-06T08:37:25.633 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-06T08:37:25.786 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-04-06T08:37:25.786 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-06T08:37:25.787 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-04-06T08:37:25.803 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-04-06T08:37:26.087 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 2 to deep-scrub 2024-04-06T08:37:26.106 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T08:37:26.403 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:37:26.403 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T08:37:26.421 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1787,"stamp":"2024-04-06T08:37:26.172395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312292,"kb_used_data":12416,"kb_used_omap":4,"kb_used_meta":299835,"kb_avail":282803228,"statfs":{"total":289910292480,"available":289590505472,"internally_reserved":0,"allocated":12713984,"data_stored":7374506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":307031378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002969"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:43.148436+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T08:34:43.148436+0000","last_peered":"2024-04-06T08:34:43.148436+0000","last_clean":"2024-04-06T08:34:43.148436+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T08:34:43.148436+0000","last_undegraded":"2024-04-06T08:34:43.148436+0000","last_fullsized":"2024-04-06T08:34:43.148436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:42:44.700586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:34:38.147644+0000","last_change":"2024-04-06T07:53:29.233161+0000","last_active":"2024-04-06T08:34:38.147644+0000","last_peered":"2024-04-06T08:34:38.147644+0000","last_clean":"2024-04-06T08:34:38.147644+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T08:34:38.147644+0000","last_undegraded":"2024-04-06T08:34:38.147644+0000","last_fullsized":"2024-04-06T08:34:38.147644+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T12:38:34.387543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152927+0000","last_change":"2024-04-06T07:53:29.235401+0000","last_active":"2024-04-06T08:35:13.152927+0000","last_peered":"2024-04-06T08:35:13.152927+0000","last_clean":"2024-04-06T08:35:13.152927+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T08:35:13.152927+0000","last_undegraded":"2024-04-06T08:35:13.152927+0000","last_fullsized":"2024-04-06T08:35:13.152927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:58:05.919002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152899+0000","last_change":"2024-04-06T07:53:29.233180+0000","last_active":"2024-04-06T08:35:13.152899+0000","last_peered":"2024-04-06T08:35:13.152899+0000","last_clean":"2024-04-06T08:35:13.152899+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T08:35:13.152899+0000","last_undegraded":"2024-04-06T08:35:13.152899+0000","last_fullsized":"2024-04-06T08:35:13.152899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T09:51:57.674391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":482,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:12.501017+0000","last_change":"2024-04-06T07:53:29.231074+0000","last_active":"2024-04-06T08:35:12.501017+0000","last_peered":"2024-04-06T08:35:12.501017+0000","last_clean":"2024-04-06T08:35:12.501017+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T08:35:12.501017+0000","last_undegraded":"2024-04-06T08:35:12.501017+0000","last_fullsized":"2024-04-06T08:35:12.501017+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:19:19.458982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:07.459490+0000","last_change":"2024-04-06T07:53:29.233823+0000","last_active":"2024-04-06T08:35:07.459490+0000","last_peered":"2024-04-06T08:35:07.459490+0000","last_clean":"2024-04-06T08:35:07.459490+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T08:35:07.459490+0000","last_undegraded":"2024-04-06T08:35:07.459490+0000","last_fullsized":"2024-04-06T08:35:07.459490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:19:37.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:42.447789+0000","last_change":"2024-04-06T07:53:29.231274+0000","last_active":"2024-04-06T08:33:42.447789+0000","last_peered":"2024-04-06T08:33:42.447789+0000","last_clean":"2024-04-06T08:33:42.447789+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T08:33:42.447789+0000","last_undegraded":"2024-04-06T08:33:42.447789+0000","last_fullsized":"2024-04-06T08:33:42.447789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T18:40:24.362133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:35:13.152872+0000","last_change":"2024-04-06T07:53:29.232232+0000","last_active":"2024-04-06T08:35:13.152872+0000","last_peered":"2024-04-06T08:35:13.152872+0000","last_clean":"2024-04-06T08:35:13.152872+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T08:35:13.152872+0000","last_undegraded":"2024-04-06T08:35:13.152872+0000","last_fullsized":"2024-04-06T08:35:13.152872+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T16:05:23.930116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":613,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:33:28.137100+0000","last_change":"2024-04-06T07:53:24.763344+0000","last_active":"2024-04-06T08:33:28.137100+0000","last_peered":"2024-04-06T08:33:28.137100+0000","last_clean":"2024-04-06T08:33:28.137100+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T08:33:28.137100+0000","last_undegraded":"2024-04-06T08:33:28.137100+0000","last_fullsized":"2024-04-06T08:33:28.137100+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:23.707131+0000","last_clean_scrub_stamp":"2024-04-06T07:53:23.707131+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:49:54.646711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97140,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274700,"statfs":{"total":96636764160,"available":96537292800,"internally_reserved":0,"allocated":3842048,"data_stored":2064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.39900000000000002},"max":{"1min":0.76400000000000001,"5min":0.76800000000000002,"15min":2.661},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63900000000000001,"15min":0.67700000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":13.782999999999999},"last":0.63500000000000001}]},{"osd":1,"last update":"Sat Apr 6 08:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.374},"max":{"1min":0.63600000000000001,"5min":0.76100000000000001,"15min":43.079999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.621},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.436},"max":{"1min":0.73599999999999999,"5min":0.754,"15min":1.3500000000000001},"last":0.48399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738901,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89528,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94282312,"statfs":{"total":96636764160,"available":96545087488,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.30299999999999999},"max":{"1min":0.67400000000000004,"5min":0.73199999999999998,"15min":6.7370000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":14.582000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46100000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.88800000000000001,"15min":2.931},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.53300000000000003,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72399999999999998,"5min":0.96599999999999997,"15min":1.288},"last":0.65800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738900,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125624,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":121278,"kb_avail":94246216,"statfs":{"total":96636764160,"available":96508125184,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124189126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.25600000000000001},"max":{"1min":0.63800000000000001,"5min":1.004,"15min":16.492999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.61399999999999999},"min":{"1min":0.52800000000000002,"5min":0.44,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":1.601,"15min":5.6680000000000001},"last":0.57399999999999995}]},{"osd":2,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.71899999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":4.0430000000000001,"15min":10.403},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":4.7320000000000002,"15min":6.5270000000000001},"last":0.75800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.425 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.426 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.426 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-06T07:53:23.707131+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:26.426 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-06T08:37:46.428 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T08:37:46.735 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:37:46.735 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T08:37:46.756 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1797,"stamp":"2024-04-06T08:37:46.177659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312280,"kb_used_data":12416,"kb_used_omap":4,"kb_used_meta":299835,"kb_avail":282803240,"statfs":{"total":289910292480,"available":289590517760,"internally_reserved":0,"allocated":12713984,"data_stored":7374506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":307031378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003285"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":483,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:36:53.167544+0000","last_change":"2024-04-06T07:53:29.235309+0000","last_active":"2024-04-06T08:36:53.167544+0000","last_peered":"2024-04-06T08:36:53.167544+0000","last_clean":"2024-04-06T08:36:53.167544+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T08:36:53.167544+0000","last_undegraded":"2024-04-06T08:36:53.167544+0000","last_fullsized":"2024-04-06T08:36:53.167544+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T07:53:28.208182+0000","last_clean_scrub_stamp":"2024-04-06T07:53:28.208182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":494,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:34.394432+0000","last_change":"2024-04-06T08:37:34.394658+0000","last_active":"2024-04-06T08:37:34.394432+0000","last_peered":"2024-04-06T08:37:34.394432+0000","last_clean":"2024-04-06T08:37:34.394432+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T08:37:34.394432+0000","last_undegraded":"2024-04-06T08:37:34.394432+0000","last_fullsized":"2024-04-06T08:37:34.394432+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:34.394382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:34.394382+0000","last_clean_scrub_stamp":"2024-04-06T08:37:34.394382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:58:36.773544+0000","scrub_duration":0.019016365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:37.323529+0000","last_change":"2024-04-06T08:37:37.323777+0000","last_active":"2024-04-06T08:37:37.323529+0000","last_peered":"2024-04-06T08:37:37.323529+0000","last_clean":"2024-04-06T08:37:37.323529+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T08:37:37.323529+0000","last_undegraded":"2024-04-06T08:37:37.323529+0000","last_fullsized":"2024-04-06T08:37:37.323529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:37.323481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:37.323481+0000","last_clean_scrub_stamp":"2024-04-06T08:37:37.323481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:02.860170+0000","scrub_duration":0.043301595999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:39.260773+0000","last_change":"2024-04-06T08:37:39.261010+0000","last_active":"2024-04-06T08:37:39.260773+0000","last_peered":"2024-04-06T08:37:39.260773+0000","last_clean":"2024-04-06T08:37:39.260773+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T08:37:39.260773+0000","last_undegraded":"2024-04-06T08:37:39.260773+0000","last_fullsized":"2024-04-06T08:37:39.260773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:39.260722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:39.260722+0000","last_clean_scrub_stamp":"2024-04-06T08:37:39.260722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:28.207649+0000","scrub_duration":0.002874288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":494,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:31.188586+0000","last_change":"2024-04-06T08:37:31.188809+0000","last_active":"2024-04-06T08:37:31.188586+0000","last_peered":"2024-04-06T08:37:31.188586+0000","last_clean":"2024-04-06T08:37:31.188586+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T08:37:31.188586+0000","last_undegraded":"2024-04-06T08:37:31.188586+0000","last_fullsized":"2024-04-06T08:37:31.188586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:31.188539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:31.188539+0000","last_clean_scrub_stamp":"2024-04-06T08:37:31.188539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:43:36.676943+0000","scrub_duration":0.015442317000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:28.208816+0000","last_change":"2024-04-06T08:37:28.209067+0000","last_active":"2024-04-06T08:37:28.208816+0000","last_peered":"2024-04-06T08:37:28.208816+0000","last_clean":"2024-04-06T08:37:28.208816+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T08:37:28.208816+0000","last_undegraded":"2024-04-06T08:37:28.208816+0000","last_fullsized":"2024-04-06T08:37:28.208816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:28.208761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:28.208761+0000","last_clean_scrub_stamp":"2024-04-06T08:37:28.208761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:26:18.141114+0000","scrub_duration":0.045427908000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":496,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:27.154095+0000","last_change":"2024-04-06T08:37:27.154095+0000","last_active":"2024-04-06T08:37:27.154095+0000","last_peered":"2024-04-06T08:37:27.154095+0000","last_clean":"2024-04-06T08:37:27.154095+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T08:37:27.154095+0000","last_undegraded":"2024-04-06T08:37:27.154095+0000","last_fullsized":"2024-04-06T08:37:27.154095+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:27.153797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:27.153797+0000","last_clean_scrub_stamp":"2024-04-06T08:37:27.153797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:29:48.373075+0000","scrub_duration":0.030689151000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:42.353858+0000","last_change":"2024-04-06T08:37:42.354106+0000","last_active":"2024-04-06T08:37:42.353858+0000","last_peered":"2024-04-06T08:37:42.353858+0000","last_clean":"2024-04-06T08:37:42.353858+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T08:37:42.353858+0000","last_undegraded":"2024-04-06T08:37:42.353858+0000","last_fullsized":"2024-04-06T08:37:42.353858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:42.353807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:42.353807+0000","last_clean_scrub_stamp":"2024-04-06T08:37:42.353807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:07:40.621516+0000","scrub_duration":0.045769595000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"467'99","reported_seq":628,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:31.383342+0000","last_change":"2024-04-06T08:37:31.383557+0000","last_active":"2024-04-06T08:37:31.383342+0000","last_peered":"2024-04-06T08:37:31.383342+0000","last_clean":"2024-04-06T08:37:31.383342+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T08:37:31.383342+0000","last_undegraded":"2024-04-06T08:37:31.383342+0000","last_fullsized":"2024-04-06T08:37:31.383342+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'99","last_scrub_stamp":"2024-04-06T08:37:31.383284+0000","last_deep_scrub":"467'99","last_deep_scrub_stamp":"2024-04-06T08:37:31.383284+0000","last_clean_scrub_stamp":"2024-04-06T08:37:31.383284+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:13:23.624428+0000","scrub_duration":0.0086274990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97136,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274704,"statfs":{"total":96636764160,"available":96537296896,"internally_reserved":0,"allocated":3842048,"data_stored":2064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.56999999999999995},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.39900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":2.661},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.68000000000000005},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":13.782999999999999},"last":0.67800000000000005}]},{"osd":1,"last update":"Sat Apr 6 08:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.66600000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.39300000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":43.079999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.47099999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.3500000000000001},"last":0.60499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738905,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89524,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94282316,"statfs":{"total":96636764160,"available":96545091584,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.30299999999999999},"max":{"1min":0.67400000000000004,"5min":0.73199999999999998,"15min":6.7370000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":14.582000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46100000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.88800000000000001,"15min":2.931},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.53300000000000003,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72399999999999998,"5min":0.96599999999999997,"15min":1.288},"last":0.68500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738904,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125620,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":121278,"kb_avail":94246220,"statfs":{"total":96636764160,"available":96508129280,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124189126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51800000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.25600000000000001},"max":{"1min":0.63800000000000001,"5min":1.004,"15min":16.492999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59299999999999997,"15min":0.61399999999999999},"min":{"1min":0.52800000000000002,"5min":0.44,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":1.601,"15min":5.6680000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Apr 6 08:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.67600000000000005,"15min":0.71899999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.33900000000000002},"max":{"1min":0.70099999999999996,"5min":4.0430000000000001,"15min":10.403},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.49299999999999999,"5min":0.48399999999999999,"15min":0.27500000000000002},"max":{"1min":0.79700000000000004,"5min":4.7320000000000002,"15min":6.5270000000000001},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:37:46.758 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-06T07:53:28.208182+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=7, tm_min=53, tm_sec=28, tm_wday=5, tm_yday=97, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=6, tm_hour=8, tm_min=37, tm_sec=23, tm_wday=5, tm_yday=97, tm_isdst=0) 2024-04-06T08:37:46.758 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-06T08:38:06.759 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-06T08:38:07.069 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-04-06T08:38:07.070 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-04-06T08:38:07.088 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1807,"stamp":"2024-04-06T08:38:06.183571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312280,"kb_used_data":12416,"kb_used_omap":4,"kb_used_meta":299835,"kb_avail":282803240,"statfs":{"total":289910292480,"available":289590517760,"internally_reserved":0,"allocated":12713984,"data_stored":7374506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":307031378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003643"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":494,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:43.336178+0000","last_change":"2024-04-06T08:37:43.336385+0000","last_active":"2024-04-06T08:37:43.336178+0000","last_peered":"2024-04-06T08:37:43.336178+0000","last_clean":"2024-04-06T08:37:43.336178+0000","last_became_active":"2024-04-06T07:53:29.234894+0000","last_became_peered":"2024-04-06T07:53:29.234894+0000","last_unstale":"2024-04-06T08:37:43.336178+0000","last_undegraded":"2024-04-06T08:37:43.336178+0000","last_fullsized":"2024-04-06T08:37:43.336178+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:43.336141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:43.336141+0000","last_clean_scrub_stamp":"2024-04-06T08:37:43.336141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:36:26.781286+0000","scrub_duration":0.064235095000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":494,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:34.394432+0000","last_change":"2024-04-06T08:37:34.394658+0000","last_active":"2024-04-06T08:37:34.394432+0000","last_peered":"2024-04-06T08:37:34.394432+0000","last_clean":"2024-04-06T08:37:34.394432+0000","last_became_active":"2024-04-06T07:53:29.232400+0000","last_became_peered":"2024-04-06T07:53:29.232400+0000","last_unstale":"2024-04-06T08:37:34.394432+0000","last_undegraded":"2024-04-06T08:37:34.394432+0000","last_fullsized":"2024-04-06T08:37:34.394432+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:34.394382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:34.394382+0000","last_clean_scrub_stamp":"2024-04-06T08:37:34.394382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:58:36.773544+0000","scrub_duration":0.019016365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:37.323529+0000","last_change":"2024-04-06T08:37:37.323777+0000","last_active":"2024-04-06T08:37:37.323529+0000","last_peered":"2024-04-06T08:37:37.323529+0000","last_clean":"2024-04-06T08:37:37.323529+0000","last_became_active":"2024-04-06T07:53:29.234969+0000","last_became_peered":"2024-04-06T07:53:29.234969+0000","last_unstale":"2024-04-06T08:37:37.323529+0000","last_undegraded":"2024-04-06T08:37:37.323529+0000","last_fullsized":"2024-04-06T08:37:37.323529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:37.323481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:37.323481+0000","last_clean_scrub_stamp":"2024-04-06T08:37:37.323481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T14:31:02.860170+0000","scrub_duration":0.043301595999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:39.260773+0000","last_change":"2024-04-06T08:37:39.261010+0000","last_active":"2024-04-06T08:37:39.260773+0000","last_peered":"2024-04-06T08:37:39.260773+0000","last_clean":"2024-04-06T08:37:39.260773+0000","last_became_active":"2024-04-06T07:53:29.232672+0000","last_became_peered":"2024-04-06T07:53:29.232672+0000","last_unstale":"2024-04-06T08:37:39.260773+0000","last_undegraded":"2024-04-06T08:37:39.260773+0000","last_fullsized":"2024-04-06T08:37:39.260773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:39.260722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:39.260722+0000","last_clean_scrub_stamp":"2024-04-06T08:37:39.260722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T17:54:28.207649+0000","scrub_duration":0.002874288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":494,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:31.188586+0000","last_change":"2024-04-06T08:37:31.188809+0000","last_active":"2024-04-06T08:37:31.188586+0000","last_peered":"2024-04-06T08:37:31.188586+0000","last_clean":"2024-04-06T08:37:31.188586+0000","last_became_active":"2024-04-06T07:53:29.230172+0000","last_became_peered":"2024-04-06T07:53:29.230172+0000","last_unstale":"2024-04-06T08:37:31.188586+0000","last_undegraded":"2024-04-06T08:37:31.188586+0000","last_fullsized":"2024-04-06T08:37:31.188586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:31.188539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:31.188539+0000","last_clean_scrub_stamp":"2024-04-06T08:37:31.188539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:43:36.676943+0000","scrub_duration":0.015442317000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:28.208816+0000","last_change":"2024-04-06T08:37:28.209067+0000","last_active":"2024-04-06T08:37:28.208816+0000","last_peered":"2024-04-06T08:37:28.208816+0000","last_clean":"2024-04-06T08:37:28.208816+0000","last_became_active":"2024-04-06T07:53:29.232700+0000","last_became_peered":"2024-04-06T07:53:29.232700+0000","last_unstale":"2024-04-06T08:37:28.208816+0000","last_undegraded":"2024-04-06T08:37:28.208816+0000","last_fullsized":"2024-04-06T08:37:28.208816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:28.208761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:28.208761+0000","last_clean_scrub_stamp":"2024-04-06T08:37:28.208761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:26:18.141114+0000","scrub_duration":0.045427908000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":496,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:27.154095+0000","last_change":"2024-04-06T08:37:27.154095+0000","last_active":"2024-04-06T08:37:27.154095+0000","last_peered":"2024-04-06T08:37:27.154095+0000","last_clean":"2024-04-06T08:37:27.154095+0000","last_became_active":"2024-04-06T07:53:29.230890+0000","last_became_peered":"2024-04-06T07:53:29.230890+0000","last_unstale":"2024-04-06T08:37:27.154095+0000","last_undegraded":"2024-04-06T08:37:27.154095+0000","last_fullsized":"2024-04-06T08:37:27.154095+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:27.153797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:27.153797+0000","last_clean_scrub_stamp":"2024-04-06T08:37:27.153797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T19:29:48.373075+0000","scrub_duration":0.030689151000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":495,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:42.353858+0000","last_change":"2024-04-06T08:37:42.354106+0000","last_active":"2024-04-06T08:37:42.353858+0000","last_peered":"2024-04-06T08:37:42.353858+0000","last_clean":"2024-04-06T08:37:42.353858+0000","last_became_active":"2024-04-06T07:53:29.231634+0000","last_became_peered":"2024-04-06T07:53:29.231634+0000","last_unstale":"2024-04-06T08:37:42.353858+0000","last_undegraded":"2024-04-06T08:37:42.353858+0000","last_fullsized":"2024-04-06T08:37:42.353858+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T08:37:42.353807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T08:37:42.353807+0000","last_clean_scrub_stamp":"2024-04-06T08:37:42.353807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T13:07:40.621516+0000","scrub_duration":0.045769595000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"467'99","reported_seq":628,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-04-06T08:37:31.383342+0000","last_change":"2024-04-06T08:37:31.383557+0000","last_active":"2024-04-06T08:37:31.383342+0000","last_peered":"2024-04-06T08:37:31.383342+0000","last_clean":"2024-04-06T08:37:31.383342+0000","last_became_active":"2024-04-06T07:53:24.760747+0000","last_became_peered":"2024-04-06T07:53:24.760747+0000","last_unstale":"2024-04-06T08:37:31.383342+0000","last_undegraded":"2024-04-06T08:37:31.383342+0000","last_fullsized":"2024-04-06T08:37:31.383342+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'99","last_scrub_stamp":"2024-04-06T08:37:31.383284+0000","last_deep_scrub":"467'99","last_deep_scrub_stamp":"2024-04-06T08:37:31.383284+0000","last_clean_scrub_stamp":"2024-04-06T08:37:31.383284+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-07T15:13:23.624428+0000","scrub_duration":0.0086274990000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97136,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94274704,"statfs":{"total":96636764160,"available":96537296896,"internally_reserved":0,"allocated":3842048,"data_stored":2064590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58799999999999997,"15min":0.56999999999999995},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.39900000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":2.661},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.68000000000000005},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.35799999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":13.782999999999999},"last":0.63800000000000001}]},{"osd":1,"last update":"Sat Apr 6 08:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55000000000000004,"15min":0.66600000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.39300000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":43.079999999999998},"last":0.499},{"interface":"front","average":{"1min":0.623,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.47099999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.3500000000000001},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738909,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89524,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94282316,"statfs":{"total":96636764160,"available":96545091584,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54700000000000004,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.30299999999999999},"max":{"1min":0.67400000000000004,"5min":0.73199999999999998,"15min":6.7370000000000001},"last":0.62},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.499,"5min":0.40000000000000002,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.65600000000000003,"15min":14.582000000000001},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Apr 6 08:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.46899999999999997,"5min":0.46100000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.88800000000000001,"15min":2.931},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.53300000000000003,"5min":0.48399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72399999999999998,"5min":0.96599999999999997,"15min":1.288},"last":0.80500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738908,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125620,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":121278,"kb_avail":94246220,"statfs":{"total":96636764160,"available":96508129280,"internally_reserved":0,"allocated":4435968,"data_stored":2654958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124189126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 6 08:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.505,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.25600000000000001},"max":{"1min":0.69899999999999995,"5min":0.71299999999999997,"15min":16.492999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.53000000000000003,"5min":0.44,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":1.601,"15min":5.6680000000000001},"last":0.498}]},{"osd":2,"last update":"Sat Apr 6 08:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.67900000000000005,"15min":0.70499999999999996},"min":{"1min":0.58499999999999996,"5min":0.50700000000000001,"15min":0.33900000000000002},"max":{"1min":0.73999999999999999,"5min":4.0430000000000001,"15min":10.403},"last":0.628},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.66500000000000004,"15min":0.67600000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.27500000000000002},"max":{"1min":0.75800000000000001,"5min":4.7320000000000002,"15min":5.6029999999999998},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T08:38:07.090 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-04-06T08:38:07.413 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1934, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2765, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2701, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-06T08:38:07.414 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-06T08:38:07.414 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-06T08:38:07.414 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-06T08:38:07.414 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:38:07.505 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-06T08:38:07.505 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-06T08:38:07.505 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:38:10.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-04-06T08:38:13.610 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-06T08:38:13.610 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-06T08:38:13.610 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:38:13.692 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-06T08:38:13.693 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-06T08:38:13.693 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-06T08:38:13.693 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:38:13.724 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-06T08:38:13.724 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-06T08:38:13.724 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-06T08:38:13.724 INFO:teuthology.orchestra.run:waiting for 300 2024-04-06T08:38:13.782 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-06T08:38:13.782 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-06T08:38:13.782 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-06T08:38:13.840 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T08:38:13.840 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-06T08:38:14.106 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T08:38:14.106 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-06T08:38:14.314 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T08:38:14.314 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-06T08:38:14.489 INFO:tasks.ceph:Archiving mon data... 2024-04-06T08:38:14.490 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/data/cluster1.mon.a.tgz 2024-04-06T08:38:14.490 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-04-06T08:38:14.495 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.8gpL1ZZjOX 2024-04-06T08:38:14.495 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.8gpL1ZZjOX 2024-04-06T08:38:14.622 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.8gpL1ZZjOX 2024-04-06T08:38:14.686 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.8gpL1ZZjOX is 337KB 2024-04-06T08:38:14.748 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.8gpL1ZZjOX 2024-04-06T08:38:14.753 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-06T08:38:14.754 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T08:38:14.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-06T08:38:14.813 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-06T08:38:14.813 DEBUG:teuthology.orchestra.run.smithi047:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-06T08:38:14.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-06T08:38:14.873 INFO:teuthology.orchestra.run.smithi047.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-06T08:38:14.877 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-06T08:38:14.882 INFO:tasks.ceph:Archiving crash dumps... 2024-04-06T08:38:14.884 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/remote/smithi047/crash 2024-04-06T08:38:14.884 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-06T08:38:14.931 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/remote/smithi129/crash 2024-04-06T08:38:14.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-06T08:38:14.985 INFO:tasks.ceph:Compressing logs... 2024-04-06T08:38:14.986 DEBUG:teuthology.orchestra.run.smithi047:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T08:38:14.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T08:38:15.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-04-06T08:41:24.212 INFO:tasks.ceph:Archiving logs... 2024-04-06T08:41:24.213 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/remote/smithi047/log 2024-04-06T08:41:24.214 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T08:41:26.869 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/remote/smithi129/log 2024-04-06T08:41:26.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T08:41:46.994 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-06T08:41:47.007 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2765, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2701, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph_052adcadd0534eddb0d73c5afbc083a84cd9dc46/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-06T08:41:47.008 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-06T08:41:47.009 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T08:41:47.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T08:41:47.076 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T08:41:47.095 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T08:41:47.095 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T08:41:47.098 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T08:41:47.108 INFO:teuthology.orchestra.run.smithi047.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T08:41:47.108 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================== 2024-04-06T08:41:47.108 INFO:teuthology.orchestra.run.smithi047.stdout:*hv01.front.sepi 44.4.53.2 2 u 29 64 377 0.080 -0.339 0.327 2024-04-06T08:41:47.108 INFO:teuthology.orchestra.run.smithi047.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 377 0.116 -1.138 0.264 2024-04-06T08:41:47.108 INFO:teuthology.orchestra.run.smithi047.stdout:+hv03.front.sepi 74.6.168.72 3 u 32 64 377 0.100 -1.067 0.308 2024-04-06T08:41:47.109 INFO:teuthology.orchestra.run.smithi047.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-06T08:41:47.110 INFO:teuthology.orchestra.run.smithi129.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T08:41:47.110 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================== 2024-04-06T08:41:47.110 INFO:teuthology.orchestra.run.smithi129.stdout:*hv01.front.sepi 44.4.53.2 2 u 40 64 377 0.090 -2.077 0.301 2024-04-06T08:41:47.111 INFO:teuthology.orchestra.run.smithi129.stdout:+hv02.front.sepi 63.231.80.2 3 u 24 64 377 0.093 -2.747 0.203 2024-04-06T08:41:47.111 INFO:teuthology.orchestra.run.smithi129.stdout:+hv03.front.sepi 74.6.168.72 3 u 16 64 377 0.119 -2.781 0.215 2024-04-06T08:41:47.111 INFO:teuthology.orchestra.run.smithi129.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-06T08:41:47.111 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T08:41:47.155 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T08:41:47.157 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T08:41:47.199 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T08:41:47.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T08:41:47.322 INFO:teuthology.task.internal:Duration was 3472.752579 seconds 2024-04-06T08:41:47.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T08:41:47.449 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T08:41:47.450 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T08:41:47.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T08:41:47.490 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T08:41:47.490 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-04-06T08:41:47.490 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T08:41:47.542 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-04-06T08:41:47.542 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T08:41:47.553 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T08:41:47.553 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T08:41:47.586 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T08:41:47.620 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T08:41:47.620 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T08:41:47.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T08:41:47.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T08:41:47.733 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T08:41:47.734 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T08:41:47.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T08:41:47.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T08:41:47.778 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T08:41:47.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T08:41:47.798 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-04-06T08:41:47.810 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-04-06T08:41:47.822 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T08:41:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T08:41:47.865 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T08:41:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T08:41:47.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T08:41:47.882 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T08:41:47.883 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/remote/smithi047 2024-04-06T08:41:47.883 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T08:41:47.929 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392/remote/smithi129 2024-04-06T08:41:47.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T08:41:47.950 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T08:41:47.950 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T08:41:47.970 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T08:41:47.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T08:41:48.009 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T08:41:48.009 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T08:41:48.123 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T08:41:48.123 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T08:41:48.125 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T08:41:48.129 INFO:teuthology.orchestra.run.smithi047.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 08:41 /home/ubuntu/cephtest 2024-04-06T08:41:48.131 INFO:teuthology.orchestra.run.smithi129.stdout: 658459 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 08:41 /home/ubuntu/cephtest 2024-04-06T08:41:48.133 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T08:41:48.181 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T08:41:48.238 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi129 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-04-06T08:41:48.238 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T08:41:48.263 DEBUG:teuthology.nuke:shortname: smithi047 2024-04-06T08:41:48.264 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T08:41:48.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 07:35:17.078725', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTHMeTm4aEYxXTztGJp/ZnBWzFTNmzWr6I2B05DKL+lPIWvSFoRnUfJ4TA4EGUpMSupoLkseaDMlft5b+aCxmc='} 2024-04-06T08:41:48.291 DEBUG:teuthology.nuke:shortname: smithi129 2024-04-06T08:41:48.291 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T08:41:48.319 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-02_22:16:03-rbd-reef-distro-default-smithi/7637392', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 07:35:17.080573', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMEK2+aO2cc17S44L4cN0L8KoZ3ZphY9k5HggEzkX1MzM4Z8bL4MNfCMihKsDKb7zUo6rjGLJC0m70T94BwUN50='} 2024-04-06T08:41:48.335 INFO:teuthology.orchestra.console.smithi047:Power off 2024-04-06T08:41:48.336 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T08:41:48.360 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2024-04-06T08:41:48.361 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:41:48.374 INFO:teuthology.orchestra.console.smithi129:Power off 2024-04-06T08:41:48.374 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T08:41:48.381 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-04-06T08:41:48.394 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-04-06T08:41:48.394 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:41:48.417 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-04-06T08:41:52.382 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:41:52.418 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:41:52.492 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-04-06T08:41:52.526 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-04-06T08:41:56.493 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:41:56.526 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:41:56.604 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2024-04-06T08:41:56.636 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-04-06T08:41:56.705 INFO:teuthology.orchestra.console.smithi047:Power off completed 2024-04-06T08:42:00.638 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T08:42:00.747 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-04-06T08:42:00.848 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-04-06T08:42:00.952 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 3472.7525794506073 failure_reason: '"2024-04-06T08:30:15.161869+0000 mon.a (mon.0) 1591 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-06T08:42:00.953 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T08:42:01.020 INFO:teuthology.run:FAIL